Corgie vehicle data PCA

Background

Four "Corgie" model vehicles were used for the experiment: a double decker bus, Cheverolet van, Saab 9000 and an Opel Manta 400 cars. This particular combination of vehicles was chosen with the expectation that the bus, van and either one of the cars would be readily distinguishable, but it would be more difficult to distinguish between the cars.

Problem Statement

The purpose is to classify a given silhouette as one of three types of vehicles (bus/car/van), using a set of features extracted from the silhouette. The vehicle may be viewed from one of many different angles.

Steps:

The points distribution for this case is as follows:

  1. Data pre-processing - Understand the data and treat missing values (Use box plot), outliers (5 points)
  2. Understanding the attributes - Find relationship between different attributes (Independent variables) and choose carefully which all attributes have to be a part of the analysis and why (5 points)
  3. Use PCA from scikit learn and elbow plot to find out reduced number of dimension (which covers more than 95% of the variance) - 10 points
  4. Use Support vector machines (find references below and research out on how to create this model) and use grid search (try C values - 0.01, 0.05, 0.5, 1 and kernel = linear, rbf) and find out the best hyper parameters and do cross validation to find the accuracy. (10 points)
In [1]:
# Import required libraries 
import pandas as pd
import seaborn as sns
import numpy as np
import math
import matplotlib.pyplot as plt 
%matplotlib inline

from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
from sklearn.impute import SimpleImputer

from sklearn.preprocessing import StandardScaler

# Import model tuning 
from sklearn.model_selection import GridSearchCV

from sklearn.metrics import classification_report, confusion_matrix  

from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
In [2]:
# To display all the variable values / full output
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"

# Ignore ipython warnings
import warnings
warnings.filterwarnings('ignore')
In [3]:
# Load dataset
df = pd.read_csv('vehicle.csv')
df.shape
Out[3]:
(846, 19)
In [4]:
df.head()
Out[4]:
compactness circularity distance_circularity radius_ratio pr.axis_aspect_ratio max.length_aspect_ratio scatter_ratio elongatedness pr.axis_rectangularity max.length_rectangularity scaled_variance scaled_variance.1 scaled_radius_of_gyration scaled_radius_of_gyration.1 skewness_about skewness_about.1 skewness_about.2 hollows_ratio class
0 95 48.0 83.0 178.0 72.0 10 162.0 42.0 20.0 159 176.0 379.0 184.0 70.0 6.0 16.0 187.0 197 van
1 91 41.0 84.0 141.0 57.0 9 149.0 45.0 19.0 143 170.0 330.0 158.0 72.0 9.0 14.0 189.0 199 van
2 104 50.0 106.0 209.0 66.0 10 207.0 32.0 23.0 158 223.0 635.0 220.0 73.0 14.0 9.0 188.0 196 car
3 93 41.0 82.0 159.0 63.0 9 144.0 46.0 19.0 143 160.0 309.0 127.0 63.0 6.0 10.0 199.0 207 van
4 85 44.0 70.0 205.0 103.0 52 149.0 45.0 19.0 144 241.0 325.0 188.0 127.0 9.0 11.0 180.0 183 bus
In [5]:
df['class'].value_counts()
Out[5]:
car    429
bus    218
van    199
Name: class, dtype: int64
In [6]:
# Encode the Class column since it has string value

le = LabelEncoder() 
df['class'] = le.fit_transform(df['class'])
df.shape
df['class'].value_counts()
Out[6]:
(846, 19)
Out[6]:
1    429
0    218
2    199
Name: class, dtype: int64

EDA - Finding missing values

In [7]:
# Check null values
df = df.replace('?', np.nan)
df.isnull().sum()
Out[7]:
compactness                    0
circularity                    5
distance_circularity           4
radius_ratio                   6
pr.axis_aspect_ratio           2
max.length_aspect_ratio        0
scatter_ratio                  1
elongatedness                  1
pr.axis_rectangularity         3
max.length_rectangularity      0
scaled_variance                3
scaled_variance.1              2
scaled_radius_of_gyration      2
scaled_radius_of_gyration.1    4
skewness_about                 6
skewness_about.1               1
skewness_about.2               1
hollows_ratio                  0
class                          0
dtype: int64
In [8]:
# There are 33 null values, it can be either removed or filled with mean value.
# Filling the null value with mean
imputer = SimpleImputer(missing_values=np.nan, strategy='median', verbose=1)

# Separate the independent and dependent values
X = df.drop('class', axis=1)
X_imputed = imputer.fit_transform(X)
X_imputed.shape
Out[8]:
(846, 18)
In [9]:
df_modified = pd.DataFrame(X_imputed, columns = X.columns)
df_modified['class'] = df[['class']] # Putting back the target value in modified dataset
df_modified.isnull().sum()
Out[9]:
compactness                    0
circularity                    0
distance_circularity           0
radius_ratio                   0
pr.axis_aspect_ratio           0
max.length_aspect_ratio        0
scatter_ratio                  0
elongatedness                  0
pr.axis_rectangularity         0
max.length_rectangularity      0
scaled_variance                0
scaled_variance.1              0
scaled_radius_of_gyration      0
scaled_radius_of_gyration.1    0
skewness_about                 0
skewness_about.1               0
skewness_about.2               0
hollows_ratio                  0
class                          0
dtype: int64

Observation: The null values are replaced with mean. there is no null in the dataset

2. Understanding the attributes

Find relationship between different attributes (Independent variables) and choose carefully which all attributes have to be a part of the analysis and why

In [10]:
df_modified.describe().transpose()
Out[10]:
count mean std min 25% 50% 75% max
compactness 846.0 93.678487 8.234474 73.0 87.00 93.0 100.00 119.0
circularity 846.0 44.823877 6.134272 33.0 40.00 44.0 49.00 59.0
distance_circularity 846.0 82.100473 15.741569 40.0 70.00 80.0 98.00 112.0
radius_ratio 846.0 168.874704 33.401356 104.0 141.00 167.0 195.00 333.0
pr.axis_aspect_ratio 846.0 61.677305 7.882188 47.0 57.00 61.0 65.00 138.0
max.length_aspect_ratio 846.0 8.567376 4.601217 2.0 7.00 8.0 10.00 55.0
scatter_ratio 846.0 168.887707 33.197710 112.0 147.00 157.0 198.00 265.0
elongatedness 846.0 40.936170 7.811882 26.0 33.00 43.0 46.00 61.0
pr.axis_rectangularity 846.0 20.580378 2.588558 17.0 19.00 20.0 23.00 29.0
max.length_rectangularity 846.0 147.998818 14.515652 118.0 137.00 146.0 159.00 188.0
scaled_variance 846.0 188.596927 31.360427 130.0 167.00 179.0 217.00 320.0
scaled_variance.1 846.0 439.314421 176.496341 184.0 318.25 363.5 586.75 1018.0
scaled_radius_of_gyration 846.0 174.706856 32.546277 109.0 149.00 173.5 198.00 268.0
scaled_radius_of_gyration.1 846.0 72.443262 7.468734 59.0 67.00 71.5 75.00 135.0
skewness_about 846.0 6.361702 4.903244 0.0 2.00 6.0 9.00 22.0
skewness_about.1 846.0 12.600473 8.930962 0.0 5.00 11.0 19.00 41.0
skewness_about.2 846.0 188.918440 6.152247 176.0 184.00 188.0 193.00 206.0
hollows_ratio 846.0 195.632388 7.438797 181.0 190.25 197.0 201.00 211.0
class 846.0 0.977541 0.702130 0.0 0.00 1.0 1.00 2.0

The describe method give the column mean, std and 3m values. The dataset is distributed equally except few columns (pr.axis_aspect_ratio, max.length_aspect_ratio, scaled_variance.1, scaled_radius_of_gyration.1, skewness_about and skewness_about.1)

In [11]:
# Finding a correlation b/w columns

plt.figure(figsize=(15,8))

sns.heatmap(df_modified.corr(),
            annot=True,
            linewidths=.5,
            center=0,
            cbar=True,
            cmap="YlGnBu")
Out[11]:
<Figure size 1080x576 with 0 Axes>
Out[11]:
<matplotlib.axes._subplots.AxesSubplot at 0x13058c8d0>

Observations from above heatmap

  • scatter_ratio is highly correlated with compactness, circularity, distance_circularity and radius_ratio
  • px.axis_rectangularity is highly correlated with compactness, circularity, distance_circularity and radius_ratio
  • max.length_rectangularity is highly correlated with circularity and distance_circularity
  • scaled_variance is highly correlated with circularity, distance_circularity and radius_ratio
  • scaled_variance.1 is highly correlated with compactness, circularity and distance_circularity
  • scaled_radius_of_gyration is highly correlated with circularity

  • skewness_about.2 is highly correlated with hollows_ratio

  • elongatedness is negatively correlated with distance_circularity, scatter_ratio, px.axis_rectangularity, scaled_variance, scaled_variance.1

In [12]:
print("skewValue of dataframe attributes")
df_modified.skew()

skewValue of dataframe attributes
Out[12]:
compactness                    0.381271
circularity                    0.264928
distance_circularity           0.108718
radius_ratio                   0.397572
pr.axis_aspect_ratio           3.835392
max.length_aspect_ratio        6.778394
scatter_ratio                  0.608710
elongatedness                  0.046951
pr.axis_rectangularity         0.774406
max.length_rectangularity      0.256359
scaled_variance                0.655598
scaled_variance.1              0.845345
scaled_radius_of_gyration      0.279910
scaled_radius_of_gyration.1    2.089979
skewness_about                 0.780813
skewness_about.1               0.689014
skewness_about.2               0.249985
hollows_ratio                 -0.226341
class                          0.031106
dtype: float64

Check outliers

In [13]:
sns.boxplot(df_modified)
Out[13]:
<matplotlib.axes._subplots.AxesSubplot at 0x132d083c8>
In [14]:
def outlier_treatment(df_in, datacolumn):
    sorted(datacolumn)
    #Q1,Q3 = np.percentile(datacolumn , [25,75])
    Q1 = df_in[datacolumn].quantile(0.25)
    Q3 = df_in[datacolumn].quantile(0.75)
    IQR = Q3 - Q1
    lower_range = Q1 - (1.5 * IQR)
    upper_range = Q3 + (1.5 * IQR)
    return lower_range,upper_range

def handle_outlier(df):
    for feature in df.columns:
        if df.dtypes[feature] != np.object and isinstance(df[feature].dtype, pd.api.types.CategoricalDtype) == False :
            lowerbound,upperbound = outlier_treatment(df, feature) #df[feature])
            print('Feature:%s, outlier length: %s' %(feature, len(df[(df[feature] < lowerbound) | (df[feature] > upperbound)])))
            if(len(df[(df[feature] < lowerbound) | (df[feature] > upperbound)]) > 0):
                df.drop(df[ (df[feature] > upperbound) | (df[feature] < lowerbound) ].index , inplace=True)
    return df
print('Outliers count:')
df_modified = handle_outlier(df_modified)
df_modified.shape

Outliers count:
Feature:compactness, outlier length: 0
Feature:circularity, outlier length: 0
Feature:distance_circularity, outlier length: 0
Feature:radius_ratio, outlier length: 3
Feature:pr.axis_aspect_ratio, outlier length: 5
Feature:max.length_aspect_ratio, outlier length: 5
Feature:scatter_ratio, outlier length: 0
Feature:elongatedness, outlier length: 0
Feature:pr.axis_rectangularity, outlier length: 0
Feature:max.length_rectangularity, outlier length: 0
Feature:scaled_variance, outlier length: 0
Feature:scaled_variance.1, outlier length: 1
Feature:scaled_radius_of_gyration, outlier length: 0
Feature:scaled_radius_of_gyration.1, outlier length: 5
Feature:skewness_about, outlier length: 12
Feature:skewness_about.1, outlier length: 1
Feature:skewness_about.2, outlier length: 0
Feature:hollows_ratio, outlier length: 0
Feature:class, outlier length: 0
Out[14]:
(814, 19)
In [15]:
plt.figure(figsize= (20,15))
plt.subplot(8,8,1)
sns.boxplot(x= df_modified['pr.axis_aspect_ratio'], color='orange')

plt.subplot(8,8,2)
sns.boxplot(x= df_modified.skewness_about, color='purple')

plt.subplot(8,8,3)
sns.boxplot(x= df_modified.scaled_variance, color='brown')
plt.subplot(8,8,4)
sns.boxplot(x= df_modified['radius_ratio'], color='red')

plt.subplot(8,8,5)
sns.boxplot(x= df_modified['scaled_radius_of_gyration.1'], color='lightblue')

plt.subplot(8,8,6)
sns.boxplot(x= df_modified['scaled_variance.1'], color='yellow')

plt.subplot(8,8,7)
sns.boxplot(x= df_modified['max.length_aspect_ratio'], color='lightblue')

plt.subplot(8,8,8)
sns.boxplot(x= df_modified['skewness_about.1'], color='pink')

plt.show()
Out[15]:
<Figure size 1440x1080 with 0 Axes>
Out[15]:
<matplotlib.axes._subplots.AxesSubplot at 0x132d67c18>
Out[15]:
<matplotlib.axes._subplots.AxesSubplot at 0x132d67c18>
Out[15]:
<matplotlib.axes._subplots.AxesSubplot at 0x132d7f0f0>
Out[15]:
<matplotlib.axes._subplots.AxesSubplot at 0x132d7f0f0>
Out[15]:
<matplotlib.axes._subplots.AxesSubplot at 0x102b75828>
Out[15]:
<matplotlib.axes._subplots.AxesSubplot at 0x102b75828>
Out[15]:
<matplotlib.axes._subplots.AxesSubplot at 0x102b9e438>
Out[15]:
<matplotlib.axes._subplots.AxesSubplot at 0x102b9e438>
Out[15]:
<matplotlib.axes._subplots.AxesSubplot at 0x13033db70>
Out[15]:
<matplotlib.axes._subplots.AxesSubplot at 0x13033db70>
Out[15]:
<matplotlib.axes._subplots.AxesSubplot at 0x130390f28>
Out[15]:
<matplotlib.axes._subplots.AxesSubplot at 0x130390f28>
Out[15]:
<matplotlib.axes._subplots.AxesSubplot at 0x1303b5a58>
Out[15]:
<matplotlib.axes._subplots.AxesSubplot at 0x1303b5a58>
Out[15]:
<matplotlib.axes._subplots.AxesSubplot at 0x133050908>
Out[15]:
<matplotlib.axes._subplots.AxesSubplot at 0x133050908>

From above image, we can evidance that outlier has been treated and removed. There is no outlier in the data now

In [16]:
# Pair plot analysis
sns.pairplot(df_modified, diag_kind='kde')
Out[16]:
<seaborn.axisgrid.PairGrid at 0x132d2be80>

Observations from Pairplot:

  • Scaled variance and scaled variance.1 are positively correlated
  • Hollow_ratio and skewness_about_2 are positively correlated
  • elongatedness and pr.axis_rectangularity are negatively correlated also
  • scatter_ratio and elongatedness have very strong negative correlated
In [18]:
def getUnique_Highly_Correlated_Columns(df):
    '''
    Get a list of duplicate columns. and column which has unique data and highly correlated in positively and negatively.
    :param df: Dataframe object
    :return: List of columns which is unwanted (contains unique data or highly correlated).
    '''
    
    unwantedColumnNames = set()
    # Iterate over all the columns in dataframe
    for x in range(df.shape[1]):
        col = df.iloc[:, x]
        x_col_name = df.columns[x]
        if(df[x_col_name].nunique() == 1 or df[x_col_name].nunique() == df.shape[0]): # Remove if All the values in columns are unique
            unwantedColumnNames.add(x_col_name)
        else:
            # Iterate over all the columns in DataFrame from (x+1)th index till end
            for y in range(x + 1, df.shape[1]):
                
                y_col_name = df.columns[y]
                if( (df.dtypes[x_col_name] != np.object) and (df.dtypes[y_col_name] != np.object)):
                    corr = df[x_col_name].corr(df[y_col_name])
                    if math.isnan(corr) == False and ~(-0.90 < corr < 0.85): # Find the correlation between the columns and remove postive and negatively correlated
                        #print(x_col_name,'--',y_col_name,' corr:', corr)
                        unwantedColumnNames.add(x_col_name)
                        break
    
    print('Unwanted columns:', list(unwantedColumnNames))
    return list(unwantedColumnNames)
df_modified = df_modified.drop(columns = getUnique_Highly_Correlated_Columns(df_modified), axis=1)
df_modified.shape

Unwanted columns: ['pr.axis_rectangularity', 'circularity', 'scatter_ratio', 'skewness_about.2', 'scaled_variance', 'elongatedness', 'distance_circularity', 'max.length_rectangularity', 'scaled_radius_of_gyration.1']
Out[18]:
(814, 10)

The following columns are highly correlated (-0.90 < corr < 0.85) with other columns. so I dropped it from the dataset.

['scaled_variance', 'scatter_ratio', 'scaled_radius_of_gyration.1', 'max.length_rectangularity', 'skewness_about.2', 'distance_circularity', 'pr.axis_rectangularity', 'circularity', 'elongatedness']

Data split

In [19]:
# Independent variables
X = df_modified.drop('class', axis=1)
# Dependent variable
y = df_modified[['class']]

Standardize the data using StandardScaler

In [20]:
sc = StandardScaler()

X_std =  sc.fit_transform(X)   
X_std.shape
Out[20]:
(814, 9)
In [21]:
X_train, X_test, y_train, y_test = train_test_split(X_std, y, test_size = 0.2, random_state = 10)
X_train.shape, X_test.shape, y_train.shape, y_test.shape
Out[21]:
((651, 9), (163, 9), (651, 1), (163, 1))

Covariance Matrix

In [22]:
# generating the covariance matrix and the eigen values for the PCA analysis
cov_matrix = np.cov(X_std.T) # the relevanat covariance matrix
print('Covariance Matrix \n%s', cov_matrix)

#generating the eigen values and the eigen vectors
e_vals, e_vecs = np.linalg.eig(cov_matrix)
print('Eigenvectors \n%s' %e_vecs)
print('\nEigenvalues \n%s' %e_vals)

Covariance Matrix 
%s [[ 1.00123001  0.74590482  0.19545948  0.49409728  0.81430389  0.58098534
   0.20185344  0.16100329  0.35826024]
 [ 0.74590482  1.00123001  0.66477545  0.46042298  0.7736483   0.55017877
   0.04063041  0.18742357  0.50331063]
 [ 0.19545948  0.66477545  1.00123001  0.14036477  0.17788025  0.14712558
  -0.0528417  -0.02865671  0.42271837]
 [ 0.49409728  0.46042298  0.14036477  1.00123001  0.45120207  0.39966846
   0.0832571   0.14099732  0.39620454]
 [ 0.81430389  0.7736483   0.17788025  0.45120207  1.00123001  0.79269652
   0.06692807  0.20272481  0.09514358]
 [ 0.58098534  0.55017877  0.14712558  0.39966846  0.79269652  1.00123001
   0.17016888 -0.05799646 -0.12581606]
 [ 0.20185344  0.04063041 -0.0528417   0.0832571   0.06692807  0.17016888
   1.00123001 -0.04531115  0.05967049]
 [ 0.16100329  0.18742357 -0.02865671  0.14099732  0.20272481 -0.05799646
  -0.04531115  1.00123001  0.19963042]
 [ 0.35826024  0.50331063  0.42271837  0.39620454  0.09514358 -0.12581606
   0.05967049  0.19963042  1.00123001]]
Eigenvectors 
[[-4.47542955e-01  1.17471817e-01  1.70453998e-02 -5.56597327e-01
  -4.35790509e-01 -5.19165658e-01  1.07620186e-01  6.90037704e-02
   4.95702232e-03]
 [-4.74411803e-01 -1.72136531e-01  7.08825200e-01  4.14002933e-01
  -5.74472525e-02 -4.94697596e-02 -1.58989247e-01 -9.38131656e-02
  -1.77890215e-01]
 [-2.39539587e-01 -4.57867761e-01 -3.27530312e-01 -2.30327634e-01
  -2.11599545e-01  3.98697586e-01 -5.20631237e-01 -5.36110178e-02
  -3.08197691e-01]
 [-3.32557885e-01 -3.82639442e-02  5.60288726e-04  4.74163335e-02
  -2.47533758e-01  5.04988905e-01  2.18775118e-01  1.34506365e-01
   7.09542973e-01]
 [-4.48679449e-01  2.86262697e-01 -5.93458214e-01  5.33634812e-01
  -3.42039903e-03 -1.54243493e-01  5.52953895e-02 -2.07575996e-01
  -9.91641441e-02]
 [-3.63473797e-01  4.64991257e-01  1.05486181e-01 -3.82030404e-01
   6.22040562e-01  2.60323523e-01 -1.72817190e-01 -9.88339939e-02
  -1.60819712e-02]
 [-8.01059660e-02  2.10643042e-01 -1.81677523e-02  1.15217781e-01
  -4.65070663e-02  1.60463376e-01  7.24896816e-02  8.81635590e-01
  -3.53779408e-01]
 [-1.11529384e-01 -2.01165359e-01  1.32764865e-02 -1.39382871e-01
   4.15987666e-02  2.90094367e-01  7.70800638e-01 -2.33769645e-01
  -4.39115902e-01]
 [-2.33200697e-01 -6.02047488e-01 -1.61782424e-01 -9.26254752e-03
   5.56686261e-01 -3.35534759e-01  1.09129001e-01  2.84052731e-01
   2.04496087e-01]]

Eigenvalues 
[3.81483581 1.5196592  0.03409422 0.1077658  0.20074307 0.47139205
 1.09597379 1.02502589 0.74158027]

Eigen values

In [23]:
# Make a set of (eigenvalue, eigenvector) pairs:

eig_pairs = [(e_vals[index], e_vecs[:,index]) for index in range(len(e_vals))]

# Sort the (eigenvalue, eigenvector) pairs from highest to lowest with respect to eigenvalue
eig_pairs.sort()

eig_pairs.reverse()
eig_pairs

# Extract the descending ordered eigenvalues and eigenvectors
eigvalues_sorted = [eig_pairs[index][0] for index in range(len(e_vals))]
eigvectors_sorted = [eig_pairs[index][1] for index in range(len(e_vecs))]

# Let's confirm our sorting worked, print out eigenvalues
print('Eigenvalues in descending order: \n%s' %eigvalues_sorted)
Out[23]:
[(3.8148358067690813,
  array([-0.44754296, -0.4744118 , -0.23953959, -0.33255789, -0.44867945,
         -0.3634738 , -0.08010597, -0.11152938, -0.2332007 ])),
 (1.5196591984037064,
  array([ 0.11747182, -0.17213653, -0.45786776, -0.03826394,  0.2862627 ,
          0.46499126,  0.21064304, -0.20116536, -0.60204749])),
 (1.0959737938241154,
  array([ 0.10762019, -0.15898925, -0.52063124,  0.21877512,  0.05529539,
         -0.17281719,  0.07248968,  0.77080064,  0.109129  ])),
 (1.025025890265232,
  array([ 0.06900377, -0.09381317, -0.05361102,  0.13450636, -0.207576  ,
         -0.09883399,  0.88163559, -0.23376964,  0.28405273])),
 (0.7415802739156597,
  array([ 0.00495702, -0.17789021, -0.30819769,  0.70954297, -0.09916414,
         -0.01608197, -0.35377941, -0.4391159 ,  0.20449609])),
 (0.4713920517526522,
  array([-0.51916566, -0.04946976,  0.39869759,  0.50498891, -0.15424349,
          0.26032352,  0.16046338,  0.29009437, -0.33553476])),
 (0.2007430732360642,
  array([-0.43579051, -0.05744725, -0.21159954, -0.24753376, -0.0034204 ,
          0.62204056, -0.04650707,  0.04159877,  0.55668626])),
 (0.10776580464582752,
  array([-0.55659733,  0.41400293, -0.23032763,  0.04741633,  0.53363481,
         -0.3820304 ,  0.11521778, -0.13938287, -0.00926255])),
 (0.03409421788876747,
  array([ 1.70453998e-02,  7.08825200e-01, -3.27530312e-01,  5.60288726e-04,
         -5.93458214e-01,  1.05486181e-01, -1.81677523e-02,  1.32764865e-02,
         -1.61782424e-01]))]
Eigenvalues in descending order: 
[3.8148358067690813, 1.5196591984037064, 1.0959737938241154, 1.025025890265232, 0.7415802739156597, 0.4713920517526522, 0.2007430732360642, 0.10776580464582752, 0.03409421788876747]

Dimentionality deduction

In [24]:
pca = PCA(n_components=9)
pca.fit(X_train)
pca.explained_variance_
pca.explained_variance_ratio_
Out[24]:
PCA(copy=True, iterated_power='auto', n_components=9, random_state=None,
    svd_solver='auto', tol=0.0, whiten=False)
Out[24]:
array([3.79971324, 1.52688579, 1.05370498, 1.03679307, 0.7724885 ,
       0.4750496 , 0.2010531 , 0.11072227, 0.03260224])
Out[24]:
array([0.42176799, 0.16948425, 0.1169612 , 0.11508398, 0.08574619,
       0.05273048, 0.02231688, 0.01229017, 0.00361885])
In [25]:
plt.bar(list(range(1,10)),pca.explained_variance_ratio_,alpha=0.5, align='center')
plt.ylabel('Variation explained')
plt.xlabel('eigen Value')
plt.show()
Out[25]:
<BarContainer object of 9 artists>
Out[25]:
Text(0, 0.5, 'Variation explained')
Out[25]:
Text(0.5, 0, 'eigen Value')
In [26]:
plt.step(list(range(1,10)),np.cumsum(pca.explained_variance_ratio_), where='mid')
plt.ylabel('Cum of variation explained')
plt.xlabel('eigen Value')
plt.show()
Out[26]:
[<matplotlib.lines.Line2D at 0x13e0178d0>]
Out[26]:
Text(0, 0.5, 'Cum of variation explained')
Out[26]:
Text(0.5, 0, 'eigen Value')

Looks like n_component=6 will be the reasonable

Dimentionality reduction

In [27]:
pca_opt = PCA(n_components=6)
pca_opt.fit(X_train)
Out[27]:
PCA(copy=True, iterated_power='auto', n_components=6, random_state=None,
    svd_solver='auto', tol=0.0, whiten=False)
In [28]:
print(pca_opt.components_)
print(pca_opt.explained_variance_ratio_)

[[ 0.4462724   0.4788661   0.24877424  0.32430417  0.44840021  0.36254409
   0.06980188  0.10605001  0.23631869]
 [ 0.11463655 -0.15648282 -0.42818423 -0.06878588  0.30124828  0.47907563
   0.16409558 -0.18152928 -0.62781696]
 [-0.1057434   0.14652294  0.50930288 -0.18434611 -0.05801095  0.14683146
  -0.12736144 -0.78867565 -0.10402631]
 [ 0.07475556 -0.08880786 -0.05487644  0.10869572 -0.1826033  -0.06044497
   0.89958971 -0.26190834  0.23841854]
 [-0.00963532  0.19329137  0.34268778 -0.73761597  0.09176784  0.01038647
   0.31441607  0.39962298 -0.18425028]
 [ 0.51305344  0.0409834  -0.41881336 -0.49153878  0.15028301 -0.25100779
  -0.16764969 -0.29287657  0.34439588]]
[0.42176799 0.16948425 0.1169612  0.11508398 0.08574619 0.05273048]
In [29]:
plt.bar(list(range(1,7)),pca_opt.explained_variance_ratio_,alpha=0.5, align='center')
plt.ylabel('Variation explained')
plt.xlabel('eigen Value')
plt.show()
Out[29]:
<BarContainer object of 6 artists>
Out[29]:
Text(0, 0.5, 'Variation explained')
Out[29]:
Text(0.5, 0, 'eigen Value')
In [30]:
# Generating pca test and train
X_train_pca = pca_opt.transform(X_train)
X_test_pca = pca_opt.transform(X_test)
X_train_pca.shape, X_test_pca.shape
Out[30]:
((651, 6), (163, 6))

Logistics Regression

In [31]:
from sklearn.linear_model import LogisticRegression

logreg = LogisticRegression()
logreg.fit(X_train, y_train)
print("Before PCA score:", logreg.score(X_test, y_test))
      
logreg.fit(X_train_pca, y_train)
print("After PCA score:", logreg.score(X_test_pca, y_test))
Out[31]:
LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
                   intercept_scaling=1, l1_ratio=None, max_iter=100,
                   multi_class='auto', n_jobs=None, penalty='l2',
                   random_state=None, solver='lbfgs', tol=0.0001, verbose=0,
                   warm_start=False)
Before PCA score: 0.9141104294478528
Out[31]:
LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
                   intercept_scaling=1, l1_ratio=None, max_iter=100,
                   multi_class='auto', n_jobs=None, penalty='l2',
                   random_state=None, solver='lbfgs', tol=0.0001, verbose=0,
                   warm_start=False)
After PCA score: 0.7975460122699386

RandomForestClassifier

In [32]:
from sklearn.ensemble import RandomForestClassifier

clf = RandomForestClassifier()
clf.fit(X_train, y_train)
print ('Before PCA score', clf.score(X_test, y_test))

clf.fit(X_train_pca, y_train)
print ('After PCA score', clf.score(X_test_pca, y_test))
Out[32]:
RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None,
                       criterion='gini', max_depth=None, max_features='auto',
                       max_leaf_nodes=None, max_samples=None,
                       min_impurity_decrease=0.0, min_impurity_split=None,
                       min_samples_leaf=1, min_samples_split=2,
                       min_weight_fraction_leaf=0.0, n_estimators=100,
                       n_jobs=None, oob_score=False, random_state=None,
                       verbose=0, warm_start=False)
Before PCA score 0.9631901840490797
Out[32]:
RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None,
                       criterion='gini', max_depth=None, max_features='auto',
                       max_leaf_nodes=None, max_samples=None,
                       min_impurity_decrease=0.0, min_impurity_split=None,
                       min_samples_leaf=1, min_samples_split=2,
                       min_weight_fraction_leaf=0.0, n_estimators=100,
                       n_jobs=None, oob_score=False, random_state=None,
                       verbose=0, warm_start=False)
After PCA score 0.8957055214723927

AdaBoostClassifier

In [33]:
from sklearn.ensemble import AdaBoostClassifier

model = AdaBoostClassifier()
model.fit(X_train, y_train)
print ('Before PCA score', model.score(X_test, y_test))

model.fit(X_train_pca, y_train)
print ('After PCA score', model.score(X_test_pca, y_test))
Out[33]:
AdaBoostClassifier(algorithm='SAMME.R', base_estimator=None, learning_rate=1.0,
                   n_estimators=50, random_state=None)
Before PCA score 0.8588957055214724
Out[33]:
AdaBoostClassifier(algorithm='SAMME.R', base_estimator=None, learning_rate=1.0,
                   n_estimators=50, random_state=None)
After PCA score 0.7791411042944786

Support vector machine (SVM)

In [34]:
from sklearn.svm import SVC
svm = SVC(C=1, gamma= 0.01, kernel='rbf')
svm.fit(X_train, y_train)
print ('Before PCA score', svm.score(X_test, y_test))

svm.fit(X_train_pca, y_train)
print ('After PCA score', svm.score(X_test_pca, y_test))
Out[34]:
SVC(C=1, break_ties=False, cache_size=200, class_weight=None, coef0=0.0,
    decision_function_shape='ovr', degree=3, gamma=0.01, kernel='rbf',
    max_iter=-1, probability=False, random_state=None, shrinking=True,
    tol=0.001, verbose=False)
Before PCA score 0.8895705521472392
Out[34]:
SVC(C=1, break_ties=False, cache_size=200, class_weight=None, coef0=0.0,
    decision_function_shape='ovr', degree=3, gamma=0.01, kernel='rbf',
    max_iter=-1, probability=False, random_state=None, shrinking=True,
    tol=0.001, verbose=False)
After PCA score 0.8404907975460123

Grid SearchCV

Use Grid Search CV to find best hyper parameters

In [35]:
# defining parameter range 
param_grid = {'C': [0.01, 0.05, 0.1, 0.5, 1, 10, 50],  
              'gamma': [1,  0.1,0.01, 0.001, 0.0001], 
              'kernel': ['linear', 'poly', 'rbf']}  
  
grid = GridSearchCV(SVC(), param_grid, refit = True, verbose = 3) 

grid.fit(X_train_pca, y_train)

Fitting 5 folds for each of 105 candidates, totalling 525 fits
[CV] C=0.01, gamma=1, kernel=linear ..................................
[CV] ...... C=0.01, gamma=1, kernel=linear, score=0.740, total=   0.0s
[CV] C=0.01, gamma=1, kernel=linear ..................................
[CV] ...... C=0.01, gamma=1, kernel=linear, score=0.723, total=   0.0s
[CV] C=0.01, gamma=1, kernel=linear ..................................
[CV] ...... C=0.01, gamma=1, kernel=linear, score=0.731, total=   0.0s
[CV] C=0.01, gamma=1, kernel=linear ..................................
[CV] ...... C=0.01, gamma=1, kernel=linear, score=0.685, total=   0.0s
[CV] C=0.01, gamma=1, kernel=linear ..................................
[CV] ...... C=0.01, gamma=1, kernel=linear, score=0.785, total=   0.0s
[CV] C=0.01, gamma=1, kernel=poly ....................................
[CV] ........ C=0.01, gamma=1, kernel=poly, score=0.832, total=   0.0s
[CV] C=0.01, gamma=1, kernel=poly ....................................
[CV] ........ C=0.01, gamma=1, kernel=poly, score=0.862, total=   0.0s
[CV] C=0.01, gamma=1, kernel=poly ....................................
[CV] ........ C=0.01, gamma=1, kernel=poly, score=0.800, total=   0.0s
[CV] C=0.01, gamma=1, kernel=poly ....................................
[CV] ........ C=0.01, gamma=1, kernel=poly, score=0.808, total=   0.0s
[CV] C=0.01, gamma=1, kernel=poly ....................................
[CV] ........ C=0.01, gamma=1, kernel=poly, score=0.877, total=   0.0s
[CV] C=0.01, gamma=1, kernel=rbf .....................................
[CV] ......... C=0.01, gamma=1, kernel=rbf, score=0.504, total=   0.0s
[CV] C=0.01, gamma=1, kernel=rbf .....................................
[CV] ......... C=0.01, gamma=1, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.01, gamma=1, kernel=rbf .....................................
[CV] ......... C=0.01, gamma=1, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.01, gamma=1, kernel=rbf .....................................
[CV] ......... C=0.01, gamma=1, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.01, gamma=1, kernel=rbf .....................................
[CV] ......... C=0.01, gamma=1, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.1, kernel=linear ................................
[CV] .... C=0.01, gamma=0.1, kernel=linear, score=0.740, total=   0.0s
[CV] C=0.01, gamma=0.1, kernel=linear ................................
[CV] .... C=0.01, gamma=0.1, kernel=linear, score=0.723, total=   0.0s
[CV] C=0.01, gamma=0.1, kernel=linear ................................
[CV] .... C=0.01, gamma=0.1, kernel=linear, score=0.731, total=   0.0s
[CV] C=0.01, gamma=0.1, kernel=linear ................................
[CV] .... C=0.01, gamma=0.1, kernel=linear, score=0.685, total=   0.0s
[CV] C=0.01, gamma=0.1, kernel=linear ................................
[CV] .... C=0.01, gamma=0.1, kernel=linear, score=0.785, total=   0.0s
[CV] C=0.01, gamma=0.1, kernel=poly ..................................
[CV] ...... C=0.01, gamma=0.1, kernel=poly, score=0.519, total=   0.0s
[CV] C=0.01, gamma=0.1, kernel=poly ..................................
[CV] ...... C=0.01, gamma=0.1, kernel=poly, score=0.523, total=   0.0s
[CV] C=0.01, gamma=0.1, kernel=poly ..................................
[CV] ...... C=0.01, gamma=0.1, kernel=poly, score=0.523, total=   0.0s
[CV] C=0.01, gamma=0.1, kernel=poly ..................................
[CV] ...... C=0.01, gamma=0.1, kernel=poly, score=0.515, total=   0.0s
[CV] C=0.01, gamma=0.1, kernel=poly ..................................
[CV] ...... C=0.01, gamma=0.1, kernel=poly, score=0.523, total=   0.0s
[CV] C=0.01, gamma=0.1, kernel=rbf ...................................
[CV] ....... C=0.01, gamma=0.1, kernel=rbf, score=0.504, total=   0.0s
[CV] C=0.01, gamma=0.1, kernel=rbf ...................................

[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.
[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s

[CV] ....... C=0.01, gamma=0.1, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.1, kernel=rbf ...................................
[CV] ....... C=0.01, gamma=0.1, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.1, kernel=rbf ...................................
[CV] ....... C=0.01, gamma=0.1, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.1, kernel=rbf ...................................
[CV] ....... C=0.01, gamma=0.1, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.01, kernel=linear ...............................
[CV] ... C=0.01, gamma=0.01, kernel=linear, score=0.740, total=   0.0s
[CV] C=0.01, gamma=0.01, kernel=linear ...............................
[CV] ... C=0.01, gamma=0.01, kernel=linear, score=0.723, total=   0.0s
[CV] C=0.01, gamma=0.01, kernel=linear ...............................
[CV] ... C=0.01, gamma=0.01, kernel=linear, score=0.731, total=   0.0s
[CV] C=0.01, gamma=0.01, kernel=linear ...............................
[CV] ... C=0.01, gamma=0.01, kernel=linear, score=0.685, total=   0.0s
[CV] C=0.01, gamma=0.01, kernel=linear ...............................
[CV] ... C=0.01, gamma=0.01, kernel=linear, score=0.785, total=   0.0s
[CV] C=0.01, gamma=0.01, kernel=poly .................................
[CV] ..... C=0.01, gamma=0.01, kernel=poly, score=0.504, total=   0.0s
[CV] C=0.01, gamma=0.01, kernel=poly .................................
[CV] ..... C=0.01, gamma=0.01, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.01, kernel=poly .................................
[CV] ..... C=0.01, gamma=0.01, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.01, kernel=poly .................................
[CV] ..... C=0.01, gamma=0.01, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.01, kernel=poly .................................
[CV] ..... C=0.01, gamma=0.01, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.01, kernel=rbf ..................................
[CV] ...... C=0.01, gamma=0.01, kernel=rbf, score=0.504, total=   0.0s
[CV] C=0.01, gamma=0.01, kernel=rbf ..................................
[CV] ...... C=0.01, gamma=0.01, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.01, kernel=rbf ..................................
[CV] ...... C=0.01, gamma=0.01, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.01, kernel=rbf ..................................
[CV] ...... C=0.01, gamma=0.01, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.01, kernel=rbf ..................................
[CV] ...... C=0.01, gamma=0.01, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.001, kernel=linear ..............................
[CV] .. C=0.01, gamma=0.001, kernel=linear, score=0.740, total=   0.0s
[CV] C=0.01, gamma=0.001, kernel=linear ..............................
[CV] .. C=0.01, gamma=0.001, kernel=linear, score=0.723, total=   0.0s
[CV] C=0.01, gamma=0.001, kernel=linear ..............................
[CV] .. C=0.01, gamma=0.001, kernel=linear, score=0.731, total=   0.0s
[CV] C=0.01, gamma=0.001, kernel=linear ..............................
[CV] .. C=0.01, gamma=0.001, kernel=linear, score=0.685, total=   0.0s
[CV] C=0.01, gamma=0.001, kernel=linear ..............................
[CV] .. C=0.01, gamma=0.001, kernel=linear, score=0.785, total=   0.0s
[CV] C=0.01, gamma=0.001, kernel=poly ................................
[CV] .... C=0.01, gamma=0.001, kernel=poly, score=0.504, total=   0.0s
[CV] C=0.01, gamma=0.001, kernel=poly ................................
[CV] .... C=0.01, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.001, kernel=poly ................................
[CV] .... C=0.01, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.001, kernel=poly ................................
[CV] .... C=0.01, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.001, kernel=poly ................................
[CV] .... C=0.01, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.001, kernel=rbf .................................
[CV] ..... C=0.01, gamma=0.001, kernel=rbf, score=0.504, total=   0.0s
[CV] C=0.01, gamma=0.001, kernel=rbf .................................
[CV] ..... C=0.01, gamma=0.001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.001, kernel=rbf .................................
[CV] ..... C=0.01, gamma=0.001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.001, kernel=rbf .................................
[CV] ..... C=0.01, gamma=0.001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.001, kernel=rbf .................................
[CV] ..... C=0.01, gamma=0.001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.0001, kernel=linear .............................
[CV] . C=0.01, gamma=0.0001, kernel=linear, score=0.740, total=   0.0s
[CV] C=0.01, gamma=0.0001, kernel=linear .............................
[CV] . C=0.01, gamma=0.0001, kernel=linear, score=0.723, total=   0.0s
[CV] C=0.01, gamma=0.0001, kernel=linear .............................
[CV] . C=0.01, gamma=0.0001, kernel=linear, score=0.731, total=   0.0s
[CV] C=0.01, gamma=0.0001, kernel=linear .............................
[CV] . C=0.01, gamma=0.0001, kernel=linear, score=0.685, total=   0.0s
[CV] C=0.01, gamma=0.0001, kernel=linear .............................
[CV] . C=0.01, gamma=0.0001, kernel=linear, score=0.785, total=   0.0s
[CV] C=0.01, gamma=0.0001, kernel=poly ...............................
[CV] ... C=0.01, gamma=0.0001, kernel=poly, score=0.504, total=   0.0s
[CV] C=0.01, gamma=0.0001, kernel=poly ...............................
[CV] ... C=0.01, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.0001, kernel=poly ...............................
[CV] ... C=0.01, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.0001, kernel=poly ...............................
[CV] ... C=0.01, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.0001, kernel=poly ...............................
[CV] ... C=0.01, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.0001, kernel=rbf ................................
[CV] .... C=0.01, gamma=0.0001, kernel=rbf, score=0.504, total=   0.0s
[CV] C=0.01, gamma=0.0001, kernel=rbf ................................
[CV] .... C=0.01, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.0001, kernel=rbf ................................
[CV] .... C=0.01, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.0001, kernel=rbf ................................
[CV] .... C=0.01, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.01, gamma=0.0001, kernel=rbf ................................
[CV] .... C=0.01, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.05, gamma=1, kernel=linear ..................................
[CV] ...... C=0.05, gamma=1, kernel=linear, score=0.763, total=   0.0s
[CV] C=0.05, gamma=1, kernel=linear ..................................
[CV] ...... C=0.05, gamma=1, kernel=linear, score=0.738, total=   0.0s
[CV] C=0.05, gamma=1, kernel=linear ..................................
[CV] ...... C=0.05, gamma=1, kernel=linear, score=0.785, total=   0.0s
[CV] C=0.05, gamma=1, kernel=linear ..................................
[CV] ...... C=0.05, gamma=1, kernel=linear, score=0.731, total=   0.0s
[CV] C=0.05, gamma=1, kernel=linear ..................................
[CV] ...... C=0.05, gamma=1, kernel=linear, score=0.831, total=   0.0s
[CV] C=0.05, gamma=1, kernel=poly ....................................
[CV] ........ C=0.05, gamma=1, kernel=poly, score=0.840, total=   0.0s
[CV] C=0.05, gamma=1, kernel=poly ....................................
[CV] ........ C=0.05, gamma=1, kernel=poly, score=0.854, total=   0.0s
[CV] C=0.05, gamma=1, kernel=poly ....................................
[CV] ........ C=0.05, gamma=1, kernel=poly, score=0.815, total=   0.0s
[CV] C=0.05, gamma=1, kernel=poly ....................................
[CV] ........ C=0.05, gamma=1, kernel=poly, score=0.885, total=   0.0s
[CV] C=0.05, gamma=1, kernel=poly ....................................
[CV] ........ C=0.05, gamma=1, kernel=poly, score=0.908, total=   0.0s
[CV] C=0.05, gamma=1, kernel=rbf .....................................
[CV] ......... C=0.05, gamma=1, kernel=rbf, score=0.504, total=   0.0s
[CV] C=0.05, gamma=1, kernel=rbf .....................................
[CV] ......... C=0.05, gamma=1, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.05, gamma=1, kernel=rbf .....................................
[CV] ......... C=0.05, gamma=1, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.05, gamma=1, kernel=rbf .....................................
[CV] ......... C=0.05, gamma=1, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.05, gamma=1, kernel=rbf .....................................
[CV] ......... C=0.05, gamma=1, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.1, kernel=linear ................................
[CV] .... C=0.05, gamma=0.1, kernel=linear, score=0.763, total=   0.0s
[CV] C=0.05, gamma=0.1, kernel=linear ................................
[CV] .... C=0.05, gamma=0.1, kernel=linear, score=0.738, total=   0.0s
[CV] C=0.05, gamma=0.1, kernel=linear ................................
[CV] .... C=0.05, gamma=0.1, kernel=linear, score=0.785, total=   0.0s
[CV] C=0.05, gamma=0.1, kernel=linear ................................
[CV] .... C=0.05, gamma=0.1, kernel=linear, score=0.731, total=   0.0s
[CV] C=0.05, gamma=0.1, kernel=linear ................................
[CV] .... C=0.05, gamma=0.1, kernel=linear, score=0.831, total=   0.0s
[CV] C=0.05, gamma=0.1, kernel=poly ..................................
[CV] ...... C=0.05, gamma=0.1, kernel=poly, score=0.527, total=   0.0s
[CV] C=0.05, gamma=0.1, kernel=poly ..................................
[CV] ...... C=0.05, gamma=0.1, kernel=poly, score=0.523, total=   0.0s
[CV] C=0.05, gamma=0.1, kernel=poly ..................................
[CV] ...... C=0.05, gamma=0.1, kernel=poly, score=0.538, total=   0.0s
[CV] C=0.05, gamma=0.1, kernel=poly ..................................
[CV] ...... C=0.05, gamma=0.1, kernel=poly, score=0.515, total=   0.0s
[CV] C=0.05, gamma=0.1, kernel=poly ..................................
[CV] ...... C=0.05, gamma=0.1, kernel=poly, score=0.538, total=   0.0s
[CV] C=0.05, gamma=0.1, kernel=rbf ...................................
[CV] ....... C=0.05, gamma=0.1, kernel=rbf, score=0.542, total=   0.0s
[CV] C=0.05, gamma=0.1, kernel=rbf ...................................
[CV] ....... C=0.05, gamma=0.1, kernel=rbf, score=0.592, total=   0.0s
[CV] C=0.05, gamma=0.1, kernel=rbf ...................................
[CV] ....... C=0.05, gamma=0.1, kernel=rbf, score=0.585, total=   0.0s
[CV] C=0.05, gamma=0.1, kernel=rbf ...................................
[CV] ....... C=0.05, gamma=0.1, kernel=rbf, score=0.585, total=   0.0s
[CV] C=0.05, gamma=0.1, kernel=rbf ...................................
[CV] ....... C=0.05, gamma=0.1, kernel=rbf, score=0.600, total=   0.0s
[CV] C=0.05, gamma=0.01, kernel=linear ...............................
[CV] ... C=0.05, gamma=0.01, kernel=linear, score=0.763, total=   0.0s
[CV] C=0.05, gamma=0.01, kernel=linear ...............................
[CV] ... C=0.05, gamma=0.01, kernel=linear, score=0.738, total=   0.0s
[CV] C=0.05, gamma=0.01, kernel=linear ...............................
[CV] ... C=0.05, gamma=0.01, kernel=linear, score=0.785, total=   0.0s
[CV] C=0.05, gamma=0.01, kernel=linear ...............................
[CV] ... C=0.05, gamma=0.01, kernel=linear, score=0.731, total=   0.0s
[CV] C=0.05, gamma=0.01, kernel=linear ...............................
[CV] ... C=0.05, gamma=0.01, kernel=linear, score=0.831, total=   0.0s
[CV] C=0.05, gamma=0.01, kernel=poly .................................
[CV] ..... C=0.05, gamma=0.01, kernel=poly, score=0.504, total=   0.0s
[CV] C=0.05, gamma=0.01, kernel=poly .................................
[CV] ..... C=0.05, gamma=0.01, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.01, kernel=poly .................................
[CV] ..... C=0.05, gamma=0.01, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.01, kernel=poly .................................
[CV] ..... C=0.05, gamma=0.01, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.01, kernel=poly .................................
[CV] ..... C=0.05, gamma=0.01, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.01, kernel=rbf ..................................
[CV] ...... C=0.05, gamma=0.01, kernel=rbf, score=0.504, total=   0.0s
[CV] C=0.05, gamma=0.01, kernel=rbf ..................................
[CV] ...... C=0.05, gamma=0.01, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.01, kernel=rbf ..................................
[CV] ...... C=0.05, gamma=0.01, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.01, kernel=rbf ..................................
[CV] ...... C=0.05, gamma=0.01, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.01, kernel=rbf ..................................
[CV] ...... C=0.05, gamma=0.01, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.001, kernel=linear ..............................
[CV] .. C=0.05, gamma=0.001, kernel=linear, score=0.763, total=   0.0s
[CV] C=0.05, gamma=0.001, kernel=linear ..............................
[CV] .. C=0.05, gamma=0.001, kernel=linear, score=0.738, total=   0.0s
[CV] C=0.05, gamma=0.001, kernel=linear ..............................
[CV] .. C=0.05, gamma=0.001, kernel=linear, score=0.785, total=   0.0s
[CV] C=0.05, gamma=0.001, kernel=linear ..............................
[CV] .. C=0.05, gamma=0.001, kernel=linear, score=0.731, total=   0.0s
[CV] C=0.05, gamma=0.001, kernel=linear ..............................
[CV] .. C=0.05, gamma=0.001, kernel=linear, score=0.831, total=   0.0s
[CV] C=0.05, gamma=0.001, kernel=poly ................................
[CV] .... C=0.05, gamma=0.001, kernel=poly, score=0.504, total=   0.0s
[CV] C=0.05, gamma=0.001, kernel=poly ................................
[CV] .... C=0.05, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.001, kernel=poly ................................
[CV] .... C=0.05, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.001, kernel=poly ................................
[CV] .... C=0.05, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.001, kernel=poly ................................
[CV] .... C=0.05, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.001, kernel=rbf .................................
[CV] ..... C=0.05, gamma=0.001, kernel=rbf, score=0.504, total=   0.0s
[CV] C=0.05, gamma=0.001, kernel=rbf .................................
[CV] ..... C=0.05, gamma=0.001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.001, kernel=rbf .................................
[CV] ..... C=0.05, gamma=0.001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.001, kernel=rbf .................................
[CV] ..... C=0.05, gamma=0.001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.001, kernel=rbf .................................
[CV] ..... C=0.05, gamma=0.001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.0001, kernel=linear .............................
[CV] . C=0.05, gamma=0.0001, kernel=linear, score=0.763, total=   0.0s
[CV] C=0.05, gamma=0.0001, kernel=linear .............................
[CV] . C=0.05, gamma=0.0001, kernel=linear, score=0.738, total=   0.0s
[CV] C=0.05, gamma=0.0001, kernel=linear .............................
[CV] . C=0.05, gamma=0.0001, kernel=linear, score=0.785, total=   0.0s
[CV] C=0.05, gamma=0.0001, kernel=linear .............................
[CV] . C=0.05, gamma=0.0001, kernel=linear, score=0.731, total=   0.0s
[CV] C=0.05, gamma=0.0001, kernel=linear .............................
[CV] . C=0.05, gamma=0.0001, kernel=linear, score=0.831, total=   0.0s
[CV] C=0.05, gamma=0.0001, kernel=poly ...............................
[CV] ... C=0.05, gamma=0.0001, kernel=poly, score=0.504, total=   0.0s
[CV] C=0.05, gamma=0.0001, kernel=poly ...............................
[CV] ... C=0.05, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.0001, kernel=poly ...............................
[CV] ... C=0.05, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.0001, kernel=poly ...............................
[CV] ... C=0.05, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.0001, kernel=poly ...............................
[CV] ... C=0.05, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.0001, kernel=rbf ................................
[CV] .... C=0.05, gamma=0.0001, kernel=rbf, score=0.504, total=   0.0s
[CV] C=0.05, gamma=0.0001, kernel=rbf ................................
[CV] .... C=0.05, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.0001, kernel=rbf ................................
[CV] .... C=0.05, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.0001, kernel=rbf ................................
[CV] .... C=0.05, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.05, gamma=0.0001, kernel=rbf ................................
[CV] .... C=0.05, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.1, gamma=1, kernel=linear ...................................
[CV] ....... C=0.1, gamma=1, kernel=linear, score=0.763, total=   0.0s
[CV] C=0.1, gamma=1, kernel=linear ...................................
[CV] ....... C=0.1, gamma=1, kernel=linear, score=0.723, total=   0.0s
[CV] C=0.1, gamma=1, kernel=linear ...................................
[CV] ....... C=0.1, gamma=1, kernel=linear, score=0.815, total=   0.0s
[CV] C=0.1, gamma=1, kernel=linear ...................................
[CV] ....... C=0.1, gamma=1, kernel=linear, score=0.738, total=   0.0s
[CV] C=0.1, gamma=1, kernel=linear ...................................
[CV] ....... C=0.1, gamma=1, kernel=linear, score=0.823, total=   0.0s
[CV] C=0.1, gamma=1, kernel=poly .....................................
[CV] ......... C=0.1, gamma=1, kernel=poly, score=0.832, total=   0.0s
[CV] C=0.1, gamma=1, kernel=poly .....................................
[CV] ......... C=0.1, gamma=1, kernel=poly, score=0.831, total=   0.0s
[CV] C=0.1, gamma=1, kernel=poly .....................................
[CV] ......... C=0.1, gamma=1, kernel=poly, score=0.831, total=   0.0s
[CV] C=0.1, gamma=1, kernel=poly .....................................
[CV] ......... C=0.1, gamma=1, kernel=poly, score=0.877, total=   0.0s
[CV] C=0.1, gamma=1, kernel=poly .....................................
[CV] ......... C=0.1, gamma=1, kernel=poly, score=0.900, total=   0.0s
[CV] C=0.1, gamma=1, kernel=rbf ......................................
[CV] .......... C=0.1, gamma=1, kernel=rbf, score=0.504, total=   0.0s
[CV] C=0.1, gamma=1, kernel=rbf ......................................
[CV] .......... C=0.1, gamma=1, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.1, gamma=1, kernel=rbf ......................................
[CV] .......... C=0.1, gamma=1, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.1, gamma=1, kernel=rbf ......................................
[CV] .......... C=0.1, gamma=1, kernel=rbf, score=0.515, total=   0.0s
[CV] C=0.1, gamma=1, kernel=rbf ......................................
[CV] .......... C=0.1, gamma=1, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.1, kernel=linear .................................
[CV] ..... C=0.1, gamma=0.1, kernel=linear, score=0.763, total=   0.0s
[CV] C=0.1, gamma=0.1, kernel=linear .................................
[CV] ..... C=0.1, gamma=0.1, kernel=linear, score=0.723, total=   0.0s
[CV] C=0.1, gamma=0.1, kernel=linear .................................
[CV] ..... C=0.1, gamma=0.1, kernel=linear, score=0.815, total=   0.0s
[CV] C=0.1, gamma=0.1, kernel=linear .................................
[CV] ..... C=0.1, gamma=0.1, kernel=linear, score=0.738, total=   0.0s
[CV] C=0.1, gamma=0.1, kernel=linear .................................
[CV] ..... C=0.1, gamma=0.1, kernel=linear, score=0.823, total=   0.0s
[CV] C=0.1, gamma=0.1, kernel=poly ...................................
[CV] ....... C=0.1, gamma=0.1, kernel=poly, score=0.534, total=   0.0s
[CV] C=0.1, gamma=0.1, kernel=poly ...................................
[CV] ....... C=0.1, gamma=0.1, kernel=poly, score=0.531, total=   0.0s
[CV] C=0.1, gamma=0.1, kernel=poly ...................................
[CV] ....... C=0.1, gamma=0.1, kernel=poly, score=0.554, total=   0.0s
[CV] C=0.1, gamma=0.1, kernel=poly ...................................
[CV] ....... C=0.1, gamma=0.1, kernel=poly, score=0.523, total=   0.0s
[CV] C=0.1, gamma=0.1, kernel=poly ...................................
[CV] ....... C=0.1, gamma=0.1, kernel=poly, score=0.554, total=   0.0s
[CV] C=0.1, gamma=0.1, kernel=rbf ....................................
[CV] ........ C=0.1, gamma=0.1, kernel=rbf, score=0.855, total=   0.0s
[CV] C=0.1, gamma=0.1, kernel=rbf ....................................
[CV] ........ C=0.1, gamma=0.1, kernel=rbf, score=0.815, total=   0.0s
[CV] C=0.1, gamma=0.1, kernel=rbf ....................................
[CV] ........ C=0.1, gamma=0.1, kernel=rbf, score=0.815, total=   0.0s
[CV] C=0.1, gamma=0.1, kernel=rbf ....................................
[CV] ........ C=0.1, gamma=0.1, kernel=rbf, score=0.800, total=   0.0s
[CV] C=0.1, gamma=0.1, kernel=rbf ....................................
[CV] ........ C=0.1, gamma=0.1, kernel=rbf, score=0.815, total=   0.0s
[CV] C=0.1, gamma=0.01, kernel=linear ................................
[CV] .... C=0.1, gamma=0.01, kernel=linear, score=0.763, total=   0.0s
[CV] C=0.1, gamma=0.01, kernel=linear ................................
[CV] .... C=0.1, gamma=0.01, kernel=linear, score=0.723, total=   0.0s
[CV] C=0.1, gamma=0.01, kernel=linear ................................
[CV] .... C=0.1, gamma=0.01, kernel=linear, score=0.815, total=   0.0s
[CV] C=0.1, gamma=0.01, kernel=linear ................................
[CV] .... C=0.1, gamma=0.01, kernel=linear, score=0.738, total=   0.0s
[CV] C=0.1, gamma=0.01, kernel=linear ................................
[CV] .... C=0.1, gamma=0.01, kernel=linear, score=0.823, total=   0.0s
[CV] C=0.1, gamma=0.01, kernel=poly ..................................
[CV] ...... C=0.1, gamma=0.01, kernel=poly, score=0.504, total=   0.0s
[CV] C=0.1, gamma=0.01, kernel=poly ..................................
[CV] ...... C=0.1, gamma=0.01, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.01, kernel=poly ..................................
[CV] ...... C=0.1, gamma=0.01, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.01, kernel=poly ..................................
[CV] ...... C=0.1, gamma=0.01, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.01, kernel=poly ..................................
[CV] ...... C=0.1, gamma=0.01, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.01, kernel=rbf ...................................
[CV] ....... C=0.1, gamma=0.01, kernel=rbf, score=0.504, total=   0.0s
[CV] C=0.1, gamma=0.01, kernel=rbf ...................................
[CV] ....... C=0.1, gamma=0.01, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.01, kernel=rbf ...................................
[CV] ....... C=0.1, gamma=0.01, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.01, kernel=rbf ...................................
[CV] ....... C=0.1, gamma=0.01, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.01, kernel=rbf ...................................
[CV] ....... C=0.1, gamma=0.01, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.001, kernel=linear ...............................
[CV] ... C=0.1, gamma=0.001, kernel=linear, score=0.763, total=   0.0s
[CV] C=0.1, gamma=0.001, kernel=linear ...............................
[CV] ... C=0.1, gamma=0.001, kernel=linear, score=0.723, total=   0.0s
[CV] C=0.1, gamma=0.001, kernel=linear ...............................
[CV] ... C=0.1, gamma=0.001, kernel=linear, score=0.815, total=   0.0s
[CV] C=0.1, gamma=0.001, kernel=linear ...............................
[CV] ... C=0.1, gamma=0.001, kernel=linear, score=0.738, total=   0.0s
[CV] C=0.1, gamma=0.001, kernel=linear ...............................
[CV] ... C=0.1, gamma=0.001, kernel=linear, score=0.823, total=   0.0s
[CV] C=0.1, gamma=0.001, kernel=poly .................................
[CV] ..... C=0.1, gamma=0.001, kernel=poly, score=0.504, total=   0.0s
[CV] C=0.1, gamma=0.001, kernel=poly .................................
[CV] ..... C=0.1, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.001, kernel=poly .................................
[CV] ..... C=0.1, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.001, kernel=poly .................................
[CV] ..... C=0.1, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.001, kernel=poly .................................
[CV] ..... C=0.1, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.001, kernel=rbf ..................................
[CV] ...... C=0.1, gamma=0.001, kernel=rbf, score=0.504, total=   0.0s
[CV] C=0.1, gamma=0.001, kernel=rbf ..................................
[CV] ...... C=0.1, gamma=0.001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.001, kernel=rbf ..................................
[CV] ...... C=0.1, gamma=0.001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.001, kernel=rbf ..................................
[CV] ...... C=0.1, gamma=0.001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.001, kernel=rbf ..................................
[CV] ...... C=0.1, gamma=0.001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.0001, kernel=linear ..............................
[CV] .. C=0.1, gamma=0.0001, kernel=linear, score=0.763, total=   0.0s
[CV] C=0.1, gamma=0.0001, kernel=linear ..............................
[CV] .. C=0.1, gamma=0.0001, kernel=linear, score=0.723, total=   0.0s
[CV] C=0.1, gamma=0.0001, kernel=linear ..............................
[CV] .. C=0.1, gamma=0.0001, kernel=linear, score=0.815, total=   0.0s
[CV] C=0.1, gamma=0.0001, kernel=linear ..............................
[CV] .. C=0.1, gamma=0.0001, kernel=linear, score=0.738, total=   0.0s
[CV] C=0.1, gamma=0.0001, kernel=linear ..............................
[CV] .. C=0.1, gamma=0.0001, kernel=linear, score=0.823, total=   0.0s
[CV] C=0.1, gamma=0.0001, kernel=poly ................................
[CV] .... C=0.1, gamma=0.0001, kernel=poly, score=0.504, total=   0.0s
[CV] C=0.1, gamma=0.0001, kernel=poly ................................
[CV] .... C=0.1, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.0001, kernel=poly ................................
[CV] .... C=0.1, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.0001, kernel=poly ................................
[CV] .... C=0.1, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.0001, kernel=poly ................................
[CV] .... C=0.1, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.0001, kernel=rbf .................................
[CV] ..... C=0.1, gamma=0.0001, kernel=rbf, score=0.504, total=   0.0s
[CV] C=0.1, gamma=0.0001, kernel=rbf .................................
[CV] ..... C=0.1, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.0001, kernel=rbf .................................
[CV] ..... C=0.1, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.0001, kernel=rbf .................................
[CV] ..... C=0.1, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.1, gamma=0.0001, kernel=rbf .................................
[CV] ..... C=0.1, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.5, gamma=1, kernel=linear ...................................
[CV] ....... C=0.5, gamma=1, kernel=linear, score=0.771, total=   0.0s
[CV] C=0.5, gamma=1, kernel=linear ...................................
[CV] ....... C=0.5, gamma=1, kernel=linear, score=0.723, total=   0.0s
[CV] C=0.5, gamma=1, kernel=linear ...................................
[CV] ....... C=0.5, gamma=1, kernel=linear, score=0.808, total=   0.0s
[CV] C=0.5, gamma=1, kernel=linear ...................................
[CV] ....... C=0.5, gamma=1, kernel=linear, score=0.746, total=   0.0s
[CV] C=0.5, gamma=1, kernel=linear ...................................
[CV] ....... C=0.5, gamma=1, kernel=linear, score=0.815, total=   0.0s
[CV] C=0.5, gamma=1, kernel=poly .....................................
[CV] ......... C=0.5, gamma=1, kernel=poly, score=0.863, total=   0.1s
[CV] C=0.5, gamma=1, kernel=poly .....................................
[CV] ......... C=0.5, gamma=1, kernel=poly, score=0.831, total=   0.1s
[CV] C=0.5, gamma=1, kernel=poly .....................................
[CV] ......... C=0.5, gamma=1, kernel=poly, score=0.838, total=   0.0s
[CV] C=0.5, gamma=1, kernel=poly .....................................
[CV] ......... C=0.5, gamma=1, kernel=poly, score=0.846, total=   0.1s
[CV] C=0.5, gamma=1, kernel=poly .....................................
[CV] ......... C=0.5, gamma=1, kernel=poly, score=0.877, total=   0.1s
[CV] C=0.5, gamma=1, kernel=rbf ......................................
[CV] .......... C=0.5, gamma=1, kernel=rbf, score=0.817, total=   0.0s
[CV] C=0.5, gamma=1, kernel=rbf ......................................
[CV] .......... C=0.5, gamma=1, kernel=rbf, score=0.785, total=   0.0s
[CV] C=0.5, gamma=1, kernel=rbf ......................................
[CV] .......... C=0.5, gamma=1, kernel=rbf, score=0.792, total=   0.0s
[CV] C=0.5, gamma=1, kernel=rbf ......................................
[CV] .......... C=0.5, gamma=1, kernel=rbf, score=0.785, total=   0.0s
[CV] C=0.5, gamma=1, kernel=rbf ......................................
[CV] .......... C=0.5, gamma=1, kernel=rbf, score=0.777, total=   0.0s
[CV] C=0.5, gamma=0.1, kernel=linear .................................
[CV] ..... C=0.5, gamma=0.1, kernel=linear, score=0.771, total=   0.0s
[CV] C=0.5, gamma=0.1, kernel=linear .................................
[CV] ..... C=0.5, gamma=0.1, kernel=linear, score=0.723, total=   0.0s
[CV] C=0.5, gamma=0.1, kernel=linear .................................
[CV] ..... C=0.5, gamma=0.1, kernel=linear, score=0.808, total=   0.0s
[CV] C=0.5, gamma=0.1, kernel=linear .................................
[CV] ..... C=0.5, gamma=0.1, kernel=linear, score=0.746, total=   0.0s
[CV] C=0.5, gamma=0.1, kernel=linear .................................
[CV] ..... C=0.5, gamma=0.1, kernel=linear, score=0.815, total=   0.0s
[CV] C=0.5, gamma=0.1, kernel=poly ...................................
[CV] ....... C=0.5, gamma=0.1, kernel=poly, score=0.695, total=   0.0s
[CV] C=0.5, gamma=0.1, kernel=poly ...................................
[CV] ....... C=0.5, gamma=0.1, kernel=poly, score=0.708, total=   0.0s
[CV] C=0.5, gamma=0.1, kernel=poly ...................................
[CV] ....... C=0.5, gamma=0.1, kernel=poly, score=0.677, total=   0.0s
[CV] C=0.5, gamma=0.1, kernel=poly ...................................
[CV] ....... C=0.5, gamma=0.1, kernel=poly, score=0.715, total=   0.0s
[CV] C=0.5, gamma=0.1, kernel=poly ...................................
[CV] ....... C=0.5, gamma=0.1, kernel=poly, score=0.777, total=   0.0s
[CV] C=0.5, gamma=0.1, kernel=rbf ....................................
[CV] ........ C=0.5, gamma=0.1, kernel=rbf, score=0.893, total=   0.0s
[CV] C=0.5, gamma=0.1, kernel=rbf ....................................
[CV] ........ C=0.5, gamma=0.1, kernel=rbf, score=0.877, total=   0.0s
[CV] C=0.5, gamma=0.1, kernel=rbf ....................................
[CV] ........ C=0.5, gamma=0.1, kernel=rbf, score=0.892, total=   0.0s
[CV] C=0.5, gamma=0.1, kernel=rbf ....................................
[CV] ........ C=0.5, gamma=0.1, kernel=rbf, score=0.838, total=   0.0s
[CV] C=0.5, gamma=0.1, kernel=rbf ....................................
[CV] ........ C=0.5, gamma=0.1, kernel=rbf, score=0.900, total=   0.0s
[CV] C=0.5, gamma=0.01, kernel=linear ................................
[CV] .... C=0.5, gamma=0.01, kernel=linear, score=0.771, total=   0.0s
[CV] C=0.5, gamma=0.01, kernel=linear ................................
[CV] .... C=0.5, gamma=0.01, kernel=linear, score=0.723, total=   0.0s
[CV] C=0.5, gamma=0.01, kernel=linear ................................
[CV] .... C=0.5, gamma=0.01, kernel=linear, score=0.808, total=   0.0s
[CV] C=0.5, gamma=0.01, kernel=linear ................................
[CV] .... C=0.5, gamma=0.01, kernel=linear, score=0.746, total=   0.0s
[CV] C=0.5, gamma=0.01, kernel=linear ................................
[CV] .... C=0.5, gamma=0.01, kernel=linear, score=0.815, total=   0.0s
[CV] C=0.5, gamma=0.01, kernel=poly ..................................
[CV] ...... C=0.5, gamma=0.01, kernel=poly, score=0.504, total=   0.0s
[CV] C=0.5, gamma=0.01, kernel=poly ..................................
[CV] ...... C=0.5, gamma=0.01, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.5, gamma=0.01, kernel=poly ..................................
[CV] ...... C=0.5, gamma=0.01, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.5, gamma=0.01, kernel=poly ..................................
[CV] ...... C=0.5, gamma=0.01, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.5, gamma=0.01, kernel=poly ..................................
[CV] ...... C=0.5, gamma=0.01, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.5, gamma=0.01, kernel=rbf ...................................
[CV] ....... C=0.5, gamma=0.01, kernel=rbf, score=0.771, total=   0.0s
[CV] C=0.5, gamma=0.01, kernel=rbf ...................................
[CV] ....... C=0.5, gamma=0.01, kernel=rbf, score=0.754, total=   0.0s
[CV] C=0.5, gamma=0.01, kernel=rbf ...................................
[CV] ....... C=0.5, gamma=0.01, kernel=rbf, score=0.738, total=   0.0s
[CV] C=0.5, gamma=0.01, kernel=rbf ...................................
[CV] ....... C=0.5, gamma=0.01, kernel=rbf, score=0.723, total=   0.0s
[CV] C=0.5, gamma=0.01, kernel=rbf ...................................
[CV] ....... C=0.5, gamma=0.01, kernel=rbf, score=0.800, total=   0.0s
[CV] C=0.5, gamma=0.001, kernel=linear ...............................
[CV] ... C=0.5, gamma=0.001, kernel=linear, score=0.771, total=   0.0s
[CV] C=0.5, gamma=0.001, kernel=linear ...............................
[CV] ... C=0.5, gamma=0.001, kernel=linear, score=0.723, total=   0.0s
[CV] C=0.5, gamma=0.001, kernel=linear ...............................
[CV] ... C=0.5, gamma=0.001, kernel=linear, score=0.808, total=   0.0s
[CV] C=0.5, gamma=0.001, kernel=linear ...............................
[CV] ... C=0.5, gamma=0.001, kernel=linear, score=0.746, total=   0.0s
[CV] C=0.5, gamma=0.001, kernel=linear ...............................
[CV] ... C=0.5, gamma=0.001, kernel=linear, score=0.815, total=   0.0s
[CV] C=0.5, gamma=0.001, kernel=poly .................................
[CV] ..... C=0.5, gamma=0.001, kernel=poly, score=0.504, total=   0.0s
[CV] C=0.5, gamma=0.001, kernel=poly .................................
[CV] ..... C=0.5, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.5, gamma=0.001, kernel=poly .................................
[CV] ..... C=0.5, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.5, gamma=0.001, kernel=poly .................................
[CV] ..... C=0.5, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.5, gamma=0.001, kernel=poly .................................
[CV] ..... C=0.5, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.5, gamma=0.001, kernel=rbf ..................................
[CV] ...... C=0.5, gamma=0.001, kernel=rbf, score=0.504, total=   0.0s
[CV] C=0.5, gamma=0.001, kernel=rbf ..................................
[CV] ...... C=0.5, gamma=0.001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.5, gamma=0.001, kernel=rbf ..................................
[CV] ...... C=0.5, gamma=0.001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.5, gamma=0.001, kernel=rbf ..................................
[CV] ...... C=0.5, gamma=0.001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.5, gamma=0.001, kernel=rbf ..................................
[CV] ...... C=0.5, gamma=0.001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.5, gamma=0.0001, kernel=linear ..............................
[CV] .. C=0.5, gamma=0.0001, kernel=linear, score=0.771, total=   0.0s
[CV] C=0.5, gamma=0.0001, kernel=linear ..............................
[CV] .. C=0.5, gamma=0.0001, kernel=linear, score=0.723, total=   0.0s
[CV] C=0.5, gamma=0.0001, kernel=linear ..............................
[CV] .. C=0.5, gamma=0.0001, kernel=linear, score=0.808, total=   0.0s
[CV] C=0.5, gamma=0.0001, kernel=linear ..............................
[CV] .. C=0.5, gamma=0.0001, kernel=linear, score=0.746, total=   0.0s
[CV] C=0.5, gamma=0.0001, kernel=linear ..............................
[CV] .. C=0.5, gamma=0.0001, kernel=linear, score=0.815, total=   0.0s
[CV] C=0.5, gamma=0.0001, kernel=poly ................................
[CV] .... C=0.5, gamma=0.0001, kernel=poly, score=0.504, total=   0.0s
[CV] C=0.5, gamma=0.0001, kernel=poly ................................
[CV] .... C=0.5, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.5, gamma=0.0001, kernel=poly ................................
[CV] .... C=0.5, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.5, gamma=0.0001, kernel=poly ................................
[CV] .... C=0.5, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.5, gamma=0.0001, kernel=poly ................................
[CV] .... C=0.5, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=0.5, gamma=0.0001, kernel=rbf .................................
[CV] ..... C=0.5, gamma=0.0001, kernel=rbf, score=0.504, total=   0.0s
[CV] C=0.5, gamma=0.0001, kernel=rbf .................................
[CV] ..... C=0.5, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.5, gamma=0.0001, kernel=rbf .................................
[CV] ..... C=0.5, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.5, gamma=0.0001, kernel=rbf .................................
[CV] ..... C=0.5, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=0.5, gamma=0.0001, kernel=rbf .................................
[CV] ..... C=0.5, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=1, gamma=1, kernel=linear .....................................
[CV] ......... C=1, gamma=1, kernel=linear, score=0.740, total=   0.0s
[CV] C=1, gamma=1, kernel=linear .....................................
[CV] ......... C=1, gamma=1, kernel=linear, score=0.723, total=   0.0s
[CV] C=1, gamma=1, kernel=linear .....................................
[CV] ......... C=1, gamma=1, kernel=linear, score=0.815, total=   0.0s
[CV] C=1, gamma=1, kernel=linear .....................................
[CV] ......... C=1, gamma=1, kernel=linear, score=0.738, total=   0.0s
[CV] C=1, gamma=1, kernel=linear .....................................
[CV] ......... C=1, gamma=1, kernel=linear, score=0.831, total=   0.0s
[CV] C=1, gamma=1, kernel=poly .......................................
[CV] ........... C=1, gamma=1, kernel=poly, score=0.855, total=   0.1s
[CV] C=1, gamma=1, kernel=poly .......................................
[CV] ........... C=1, gamma=1, kernel=poly, score=0.838, total=   0.1s
[CV] C=1, gamma=1, kernel=poly .......................................
[CV] ........... C=1, gamma=1, kernel=poly, score=0.846, total=   0.1s
[CV] C=1, gamma=1, kernel=poly .......................................
[CV] ........... C=1, gamma=1, kernel=poly, score=0.831, total=   0.2s
[CV] C=1, gamma=1, kernel=poly .......................................
[CV] ........... C=1, gamma=1, kernel=poly, score=0.877, total=   0.2s
[CV] C=1, gamma=1, kernel=rbf ........................................
[CV] ............ C=1, gamma=1, kernel=rbf, score=0.878, total=   0.0s
[CV] C=1, gamma=1, kernel=rbf ........................................
[CV] ............ C=1, gamma=1, kernel=rbf, score=0.838, total=   0.0s
[CV] C=1, gamma=1, kernel=rbf ........................................
[CV] ............ C=1, gamma=1, kernel=rbf, score=0.854, total=   0.0s
[CV] C=1, gamma=1, kernel=rbf ........................................
[CV] ............ C=1, gamma=1, kernel=rbf, score=0.808, total=   0.0s
[CV] C=1, gamma=1, kernel=rbf ........................................
[CV] ............ C=1, gamma=1, kernel=rbf, score=0.831, total=   0.0s
[CV] C=1, gamma=0.1, kernel=linear ...................................
[CV] ....... C=1, gamma=0.1, kernel=linear, score=0.740, total=   0.0s
[CV] C=1, gamma=0.1, kernel=linear ...................................
[CV] ....... C=1, gamma=0.1, kernel=linear, score=0.723, total=   0.0s
[CV] C=1, gamma=0.1, kernel=linear ...................................
[CV] ....... C=1, gamma=0.1, kernel=linear, score=0.815, total=   0.0s
[CV] C=1, gamma=0.1, kernel=linear ...................................
[CV] ....... C=1, gamma=0.1, kernel=linear, score=0.738, total=   0.0s
[CV] C=1, gamma=0.1, kernel=linear ...................................
[CV] ....... C=1, gamma=0.1, kernel=linear, score=0.831, total=   0.0s
[CV] C=1, gamma=0.1, kernel=poly .....................................
[CV] ......... C=1, gamma=0.1, kernel=poly, score=0.763, total=   0.0s
[CV] C=1, gamma=0.1, kernel=poly .....................................
[CV] ......... C=1, gamma=0.1, kernel=poly, score=0.762, total=   0.0s
[CV] C=1, gamma=0.1, kernel=poly .....................................
[CV] ......... C=1, gamma=0.1, kernel=poly, score=0.731, total=   0.0s
[CV] C=1, gamma=0.1, kernel=poly .....................................
[CV] ......... C=1, gamma=0.1, kernel=poly, score=0.769, total=   0.0s
[CV] C=1, gamma=0.1, kernel=poly .....................................
[CV] ......... C=1, gamma=0.1, kernel=poly, score=0.823, total=   0.0s
[CV] C=1, gamma=0.1, kernel=rbf ......................................
[CV] .......... C=1, gamma=0.1, kernel=rbf, score=0.885, total=   0.0s
[CV] C=1, gamma=0.1, kernel=rbf ......................................
[CV] .......... C=1, gamma=0.1, kernel=rbf, score=0.885, total=   0.0s
[CV] C=1, gamma=0.1, kernel=rbf ......................................
[CV] .......... C=1, gamma=0.1, kernel=rbf, score=0.900, total=   0.0s
[CV] C=1, gamma=0.1, kernel=rbf ......................................
[CV] .......... C=1, gamma=0.1, kernel=rbf, score=0.862, total=   0.0s
[CV] C=1, gamma=0.1, kernel=rbf ......................................
[CV] .......... C=1, gamma=0.1, kernel=rbf, score=0.908, total=   0.0s
[CV] C=1, gamma=0.01, kernel=linear ..................................
[CV] ...... C=1, gamma=0.01, kernel=linear, score=0.740, total=   0.0s
[CV] C=1, gamma=0.01, kernel=linear ..................................
[CV] ...... C=1, gamma=0.01, kernel=linear, score=0.723, total=   0.0s
[CV] C=1, gamma=0.01, kernel=linear ..................................
[CV] ...... C=1, gamma=0.01, kernel=linear, score=0.815, total=   0.0s
[CV] C=1, gamma=0.01, kernel=linear ..................................
[CV] ...... C=1, gamma=0.01, kernel=linear, score=0.738, total=   0.0s
[CV] C=1, gamma=0.01, kernel=linear ..................................
[CV] ...... C=1, gamma=0.01, kernel=linear, score=0.831, total=   0.0s
[CV] C=1, gamma=0.01, kernel=poly ....................................
[CV] ........ C=1, gamma=0.01, kernel=poly, score=0.504, total=   0.0s
[CV] C=1, gamma=0.01, kernel=poly ....................................
[CV] ........ C=1, gamma=0.01, kernel=poly, score=0.500, total=   0.0s
[CV] C=1, gamma=0.01, kernel=poly ....................................
[CV] ........ C=1, gamma=0.01, kernel=poly, score=0.500, total=   0.0s
[CV] C=1, gamma=0.01, kernel=poly ....................................
[CV] ........ C=1, gamma=0.01, kernel=poly, score=0.500, total=   0.0s
[CV] C=1, gamma=0.01, kernel=poly ....................................
[CV] ........ C=1, gamma=0.01, kernel=poly, score=0.500, total=   0.0s
[CV] C=1, gamma=0.01, kernel=rbf .....................................
[CV] ......... C=1, gamma=0.01, kernel=rbf, score=0.786, total=   0.0s
[CV] C=1, gamma=0.01, kernel=rbf .....................................
[CV] ......... C=1, gamma=0.01, kernel=rbf, score=0.785, total=   0.0s
[CV] C=1, gamma=0.01, kernel=rbf .....................................
[CV] ......... C=1, gamma=0.01, kernel=rbf, score=0.815, total=   0.0s
[CV] C=1, gamma=0.01, kernel=rbf .....................................
[CV] ......... C=1, gamma=0.01, kernel=rbf, score=0.769, total=   0.0s
[CV] C=1, gamma=0.01, kernel=rbf .....................................
[CV] ......... C=1, gamma=0.01, kernel=rbf, score=0.831, total=   0.0s
[CV] C=1, gamma=0.001, kernel=linear .................................
[CV] ..... C=1, gamma=0.001, kernel=linear, score=0.740, total=   0.0s
[CV] C=1, gamma=0.001, kernel=linear .................................
[CV] ..... C=1, gamma=0.001, kernel=linear, score=0.723, total=   0.0s
[CV] C=1, gamma=0.001, kernel=linear .................................
[CV] ..... C=1, gamma=0.001, kernel=linear, score=0.815, total=   0.0s
[CV] C=1, gamma=0.001, kernel=linear .................................
[CV] ..... C=1, gamma=0.001, kernel=linear, score=0.738, total=   0.0s
[CV] C=1, gamma=0.001, kernel=linear .................................
[CV] ..... C=1, gamma=0.001, kernel=linear, score=0.831, total=   0.0s
[CV] C=1, gamma=0.001, kernel=poly ...................................
[CV] ....... C=1, gamma=0.001, kernel=poly, score=0.504, total=   0.0s
[CV] C=1, gamma=0.001, kernel=poly ...................................
[CV] ....... C=1, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=1, gamma=0.001, kernel=poly ...................................
[CV] ....... C=1, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=1, gamma=0.001, kernel=poly ...................................
[CV] ....... C=1, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=1, gamma=0.001, kernel=poly ...................................
[CV] ....... C=1, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=1, gamma=0.001, kernel=rbf ....................................
[CV] ........ C=1, gamma=0.001, kernel=rbf, score=0.504, total=   0.0s
[CV] C=1, gamma=0.001, kernel=rbf ....................................
[CV] ........ C=1, gamma=0.001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=1, gamma=0.001, kernel=rbf ....................................
[CV] ........ C=1, gamma=0.001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=1, gamma=0.001, kernel=rbf ....................................
[CV] ........ C=1, gamma=0.001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=1, gamma=0.001, kernel=rbf ....................................
[CV] ........ C=1, gamma=0.001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=1, gamma=0.0001, kernel=linear ................................
[CV] .... C=1, gamma=0.0001, kernel=linear, score=0.740, total=   0.0s
[CV] C=1, gamma=0.0001, kernel=linear ................................
[CV] .... C=1, gamma=0.0001, kernel=linear, score=0.723, total=   0.0s
[CV] C=1, gamma=0.0001, kernel=linear ................................
[CV] .... C=1, gamma=0.0001, kernel=linear, score=0.815, total=   0.0s
[CV] C=1, gamma=0.0001, kernel=linear ................................
[CV] .... C=1, gamma=0.0001, kernel=linear, score=0.738, total=   0.0s
[CV] C=1, gamma=0.0001, kernel=linear ................................
[CV] .... C=1, gamma=0.0001, kernel=linear, score=0.831, total=   0.0s
[CV] C=1, gamma=0.0001, kernel=poly ..................................
[CV] ...... C=1, gamma=0.0001, kernel=poly, score=0.504, total=   0.0s
[CV] C=1, gamma=0.0001, kernel=poly ..................................
[CV] ...... C=1, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=1, gamma=0.0001, kernel=poly ..................................
[CV] ...... C=1, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=1, gamma=0.0001, kernel=poly ..................................
[CV] ...... C=1, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=1, gamma=0.0001, kernel=poly ..................................
[CV] ...... C=1, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=1, gamma=0.0001, kernel=rbf ...................................
[CV] ....... C=1, gamma=0.0001, kernel=rbf, score=0.504, total=   0.0s
[CV] C=1, gamma=0.0001, kernel=rbf ...................................
[CV] ....... C=1, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=1, gamma=0.0001, kernel=rbf ...................................
[CV] ....... C=1, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=1, gamma=0.0001, kernel=rbf ...................................
[CV] ....... C=1, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=1, gamma=0.0001, kernel=rbf ...................................
[CV] ....... C=1, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=10, gamma=1, kernel=linear ....................................
[CV] ........ C=10, gamma=1, kernel=linear, score=0.763, total=   0.0s
[CV] C=10, gamma=1, kernel=linear ....................................
[CV] ........ C=10, gamma=1, kernel=linear, score=0.731, total=   0.0s
[CV] C=10, gamma=1, kernel=linear ....................................
[CV] ........ C=10, gamma=1, kernel=linear, score=0.815, total=   0.0s
[CV] C=10, gamma=1, kernel=linear ....................................
[CV] ........ C=10, gamma=1, kernel=linear, score=0.738, total=   0.0s
[CV] C=10, gamma=1, kernel=linear ....................................
[CV] ........ C=10, gamma=1, kernel=linear, score=0.815, total=   0.0s
[CV] C=10, gamma=1, kernel=poly ......................................
[CV] .......... C=10, gamma=1, kernel=poly, score=0.817, total=   0.8s
[CV] C=10, gamma=1, kernel=poly ......................................
[CV] .......... C=10, gamma=1, kernel=poly, score=0.808, total=   0.6s
[CV] C=10, gamma=1, kernel=poly ......................................
[CV] .......... C=10, gamma=1, kernel=poly, score=0.838, total=   0.5s
[CV] C=10, gamma=1, kernel=poly ......................................
[CV] .......... C=10, gamma=1, kernel=poly, score=0.823, total=   0.6s
[CV] C=10, gamma=1, kernel=poly ......................................
[CV] .......... C=10, gamma=1, kernel=poly, score=0.869, total=   1.2s
[CV] C=10, gamma=1, kernel=rbf .......................................
[CV] ........... C=10, gamma=1, kernel=rbf, score=0.832, total=   0.0s
[CV] C=10, gamma=1, kernel=rbf .......................................
[CV] ........... C=10, gamma=1, kernel=rbf, score=0.823, total=   0.0s
[CV] C=10, gamma=1, kernel=rbf .......................................
[CV] ........... C=10, gamma=1, kernel=rbf, score=0.831, total=   0.0s
[CV] C=10, gamma=1, kernel=rbf .......................................
[CV] ........... C=10, gamma=1, kernel=rbf, score=0.831, total=   0.0s
[CV] C=10, gamma=1, kernel=rbf .......................................
[CV] ........... C=10, gamma=1, kernel=rbf, score=0.838, total=   0.0s
[CV] C=10, gamma=0.1, kernel=linear ..................................
[CV] ...... C=10, gamma=0.1, kernel=linear, score=0.763, total=   0.0s
[CV] C=10, gamma=0.1, kernel=linear ..................................
[CV] ...... C=10, gamma=0.1, kernel=linear, score=0.731, total=   0.0s
[CV] C=10, gamma=0.1, kernel=linear ..................................
[CV] ...... C=10, gamma=0.1, kernel=linear, score=0.815, total=   0.0s
[CV] C=10, gamma=0.1, kernel=linear ..................................
[CV] ...... C=10, gamma=0.1, kernel=linear, score=0.738, total=   0.0s
[CV] C=10, gamma=0.1, kernel=linear ..................................
[CV] ...... C=10, gamma=0.1, kernel=linear, score=0.815, total=   0.0s
[CV] C=10, gamma=0.1, kernel=poly ....................................
[CV] ........ C=10, gamma=0.1, kernel=poly, score=0.832, total=   0.0s
[CV] C=10, gamma=0.1, kernel=poly ....................................
[CV] ........ C=10, gamma=0.1, kernel=poly, score=0.862, total=   0.0s
[CV] C=10, gamma=0.1, kernel=poly ....................................
[CV] ........ C=10, gamma=0.1, kernel=poly, score=0.800, total=   0.0s
[CV] C=10, gamma=0.1, kernel=poly ....................................
[CV] ........ C=10, gamma=0.1, kernel=poly, score=0.808, total=   0.0s
[CV] C=10, gamma=0.1, kernel=poly ....................................
[CV] ........ C=10, gamma=0.1, kernel=poly, score=0.877, total=   0.0s
[CV] C=10, gamma=0.1, kernel=rbf .....................................
[CV] ......... C=10, gamma=0.1, kernel=rbf, score=0.901, total=   0.0s
[CV] C=10, gamma=0.1, kernel=rbf .....................................
[CV] ......... C=10, gamma=0.1, kernel=rbf, score=0.877, total=   0.0s
[CV] C=10, gamma=0.1, kernel=rbf .....................................
[CV] ......... C=10, gamma=0.1, kernel=rbf, score=0.915, total=   0.0s
[CV] C=10, gamma=0.1, kernel=rbf .....................................
[CV] ......... C=10, gamma=0.1, kernel=rbf, score=0.869, total=   0.0s
[CV] C=10, gamma=0.1, kernel=rbf .....................................
[CV] ......... C=10, gamma=0.1, kernel=rbf, score=0.923, total=   0.0s
[CV] C=10, gamma=0.01, kernel=linear .................................
[CV] ..... C=10, gamma=0.01, kernel=linear, score=0.763, total=   0.0s
[CV] C=10, gamma=0.01, kernel=linear .................................
[CV] ..... C=10, gamma=0.01, kernel=linear, score=0.731, total=   0.0s
[CV] C=10, gamma=0.01, kernel=linear .................................
[CV] ..... C=10, gamma=0.01, kernel=linear, score=0.815, total=   0.0s
[CV] C=10, gamma=0.01, kernel=linear .................................
[CV] ..... C=10, gamma=0.01, kernel=linear, score=0.738, total=   0.0s
[CV] C=10, gamma=0.01, kernel=linear .................................
[CV] ..... C=10, gamma=0.01, kernel=linear, score=0.815, total=   0.0s
[CV] C=10, gamma=0.01, kernel=poly ...................................
[CV] ....... C=10, gamma=0.01, kernel=poly, score=0.519, total=   0.0s
[CV] C=10, gamma=0.01, kernel=poly ...................................
[CV] ....... C=10, gamma=0.01, kernel=poly, score=0.523, total=   0.0s
[CV] C=10, gamma=0.01, kernel=poly ...................................
[CV] ....... C=10, gamma=0.01, kernel=poly, score=0.523, total=   0.0s
[CV] C=10, gamma=0.01, kernel=poly ...................................
[CV] ....... C=10, gamma=0.01, kernel=poly, score=0.515, total=   0.0s
[CV] C=10, gamma=0.01, kernel=poly ...................................
[CV] ....... C=10, gamma=0.01, kernel=poly, score=0.523, total=   0.0s
[CV] C=10, gamma=0.01, kernel=rbf ....................................
[CV] ........ C=10, gamma=0.01, kernel=rbf, score=0.855, total=   0.0s
[CV] C=10, gamma=0.01, kernel=rbf ....................................
[CV] ........ C=10, gamma=0.01, kernel=rbf, score=0.823, total=   0.0s
[CV] C=10, gamma=0.01, kernel=rbf ....................................
[CV] ........ C=10, gamma=0.01, kernel=rbf, score=0.862, total=   0.0s
[CV] C=10, gamma=0.01, kernel=rbf ....................................
[CV] ........ C=10, gamma=0.01, kernel=rbf, score=0.838, total=   0.0s
[CV] C=10, gamma=0.01, kernel=rbf ....................................
[CV] ........ C=10, gamma=0.01, kernel=rbf, score=0.869, total=   0.0s
[CV] C=10, gamma=0.001, kernel=linear ................................
[CV] .... C=10, gamma=0.001, kernel=linear, score=0.763, total=   0.0s
[CV] C=10, gamma=0.001, kernel=linear ................................
[CV] .... C=10, gamma=0.001, kernel=linear, score=0.731, total=   0.0s
[CV] C=10, gamma=0.001, kernel=linear ................................
[CV] .... C=10, gamma=0.001, kernel=linear, score=0.815, total=   0.0s
[CV] C=10, gamma=0.001, kernel=linear ................................
[CV] .... C=10, gamma=0.001, kernel=linear, score=0.738, total=   0.0s
[CV] C=10, gamma=0.001, kernel=linear ................................
[CV] .... C=10, gamma=0.001, kernel=linear, score=0.815, total=   0.0s
[CV] C=10, gamma=0.001, kernel=poly ..................................
[CV] ...... C=10, gamma=0.001, kernel=poly, score=0.504, total=   0.0s
[CV] C=10, gamma=0.001, kernel=poly ..................................
[CV] ...... C=10, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=10, gamma=0.001, kernel=poly ..................................
[CV] ...... C=10, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=10, gamma=0.001, kernel=poly ..................................
[CV] ...... C=10, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=10, gamma=0.001, kernel=poly ..................................
[CV] ...... C=10, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=10, gamma=0.001, kernel=rbf ...................................
[CV] ....... C=10, gamma=0.001, kernel=rbf, score=0.756, total=   0.0s
[CV] C=10, gamma=0.001, kernel=rbf ...................................
[CV] ....... C=10, gamma=0.001, kernel=rbf, score=0.738, total=   0.0s
[CV] C=10, gamma=0.001, kernel=rbf ...................................
[CV] ....... C=10, gamma=0.001, kernel=rbf, score=0.815, total=   0.0s
[CV] C=10, gamma=0.001, kernel=rbf ...................................
[CV] ....... C=10, gamma=0.001, kernel=rbf, score=0.715, total=   0.0s
[CV] C=10, gamma=0.001, kernel=rbf ...................................
[CV] ....... C=10, gamma=0.001, kernel=rbf, score=0.831, total=   0.0s
[CV] C=10, gamma=0.0001, kernel=linear ...............................
[CV] ... C=10, gamma=0.0001, kernel=linear, score=0.763, total=   0.0s
[CV] C=10, gamma=0.0001, kernel=linear ...............................
[CV] ... C=10, gamma=0.0001, kernel=linear, score=0.731, total=   0.0s
[CV] C=10, gamma=0.0001, kernel=linear ...............................
[CV] ... C=10, gamma=0.0001, kernel=linear, score=0.815, total=   0.0s
[CV] C=10, gamma=0.0001, kernel=linear ...............................
[CV] ... C=10, gamma=0.0001, kernel=linear, score=0.738, total=   0.0s
[CV] C=10, gamma=0.0001, kernel=linear ...............................
[CV] ... C=10, gamma=0.0001, kernel=linear, score=0.815, total=   0.0s
[CV] C=10, gamma=0.0001, kernel=poly .................................
[CV] ..... C=10, gamma=0.0001, kernel=poly, score=0.504, total=   0.0s
[CV] C=10, gamma=0.0001, kernel=poly .................................
[CV] ..... C=10, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=10, gamma=0.0001, kernel=poly .................................
[CV] ..... C=10, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=10, gamma=0.0001, kernel=poly .................................
[CV] ..... C=10, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=10, gamma=0.0001, kernel=poly .................................
[CV] ..... C=10, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=10, gamma=0.0001, kernel=rbf ..................................
[CV] ...... C=10, gamma=0.0001, kernel=rbf, score=0.504, total=   0.0s
[CV] C=10, gamma=0.0001, kernel=rbf ..................................
[CV] ...... C=10, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=10, gamma=0.0001, kernel=rbf ..................................
[CV] ...... C=10, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=10, gamma=0.0001, kernel=rbf ..................................
[CV] ...... C=10, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=10, gamma=0.0001, kernel=rbf ..................................
[CV] ...... C=10, gamma=0.0001, kernel=rbf, score=0.500, total=   0.0s
[CV] C=50, gamma=1, kernel=linear ....................................
[CV] ........ C=50, gamma=1, kernel=linear, score=0.771, total=   0.1s
[CV] C=50, gamma=1, kernel=linear ....................................
[CV] ........ C=50, gamma=1, kernel=linear, score=0.723, total=   0.0s
[CV] C=50, gamma=1, kernel=linear ....................................
[CV] ........ C=50, gamma=1, kernel=linear, score=0.815, total=   0.1s
[CV] C=50, gamma=1, kernel=linear ....................................
[CV] ........ C=50, gamma=1, kernel=linear, score=0.738, total=   0.1s
[CV] C=50, gamma=1, kernel=linear ....................................
[CV] ........ C=50, gamma=1, kernel=linear, score=0.815, total=   0.1s
[CV] C=50, gamma=1, kernel=poly ......................................
[CV] .......... C=50, gamma=1, kernel=poly, score=0.809, total=   3.7s
[CV] C=50, gamma=1, kernel=poly ......................................
[CV] .......... C=50, gamma=1, kernel=poly, score=0.800, total=   3.0s
[CV] C=50, gamma=1, kernel=poly ......................................
[CV] .......... C=50, gamma=1, kernel=poly, score=0.815, total=   4.5s
[CV] C=50, gamma=1, kernel=poly ......................................
[CV] .......... C=50, gamma=1, kernel=poly, score=0.815, total=   1.2s
[CV] C=50, gamma=1, kernel=poly ......................................
[CV] .......... C=50, gamma=1, kernel=poly, score=0.862, total=   8.7s
[CV] C=50, gamma=1, kernel=rbf .......................................
[CV] ........... C=50, gamma=1, kernel=rbf, score=0.832, total=   0.0s
[CV] C=50, gamma=1, kernel=rbf .......................................
[CV] ........... C=50, gamma=1, kernel=rbf, score=0.823, total=   0.0s
[CV] C=50, gamma=1, kernel=rbf .......................................
[CV] ........... C=50, gamma=1, kernel=rbf, score=0.823, total=   0.0s
[CV] C=50, gamma=1, kernel=rbf .......................................
[CV] ........... C=50, gamma=1, kernel=rbf, score=0.831, total=   0.0s
[CV] C=50, gamma=1, kernel=rbf .......................................
[CV] ........... C=50, gamma=1, kernel=rbf, score=0.838, total=   0.0s
[CV] C=50, gamma=0.1, kernel=linear ..................................
[CV] ...... C=50, gamma=0.1, kernel=linear, score=0.771, total=   0.1s
[CV] C=50, gamma=0.1, kernel=linear ..................................
[CV] ...... C=50, gamma=0.1, kernel=linear, score=0.723, total=   0.0s
[CV] C=50, gamma=0.1, kernel=linear ..................................
[CV] ...... C=50, gamma=0.1, kernel=linear, score=0.815, total=   0.1s
[CV] C=50, gamma=0.1, kernel=linear ..................................
[CV] ...... C=50, gamma=0.1, kernel=linear, score=0.738, total=   0.1s
[CV] C=50, gamma=0.1, kernel=linear ..................................
[CV] ...... C=50, gamma=0.1, kernel=linear, score=0.815, total=   0.1s
[CV] C=50, gamma=0.1, kernel=poly ....................................
[CV] ........ C=50, gamma=0.1, kernel=poly, score=0.840, total=   0.0s
[CV] C=50, gamma=0.1, kernel=poly ....................................
[CV] ........ C=50, gamma=0.1, kernel=poly, score=0.854, total=   0.0s
[CV] C=50, gamma=0.1, kernel=poly ....................................
[CV] ........ C=50, gamma=0.1, kernel=poly, score=0.815, total=   0.0s
[CV] C=50, gamma=0.1, kernel=poly ....................................
[CV] ........ C=50, gamma=0.1, kernel=poly, score=0.885, total=   0.0s
[CV] C=50, gamma=0.1, kernel=poly ....................................
[CV] ........ C=50, gamma=0.1, kernel=poly, score=0.908, total=   0.0s
[CV] C=50, gamma=0.1, kernel=rbf .....................................
[CV] ......... C=50, gamma=0.1, kernel=rbf, score=0.885, total=   0.0s
[CV] C=50, gamma=0.1, kernel=rbf .....................................
[CV] ......... C=50, gamma=0.1, kernel=rbf, score=0.869, total=   0.0s
[CV] C=50, gamma=0.1, kernel=rbf .....................................
[CV] ......... C=50, gamma=0.1, kernel=rbf, score=0.900, total=   0.0s
[CV] C=50, gamma=0.1, kernel=rbf .....................................
[CV] ......... C=50, gamma=0.1, kernel=rbf, score=0.862, total=   0.0s
[CV] C=50, gamma=0.1, kernel=rbf .....................................
[CV] ......... C=50, gamma=0.1, kernel=rbf, score=0.892, total=   0.0s
[CV] C=50, gamma=0.01, kernel=linear .................................
[CV] ..... C=50, gamma=0.01, kernel=linear, score=0.771, total=   0.1s
[CV] C=50, gamma=0.01, kernel=linear .................................
[CV] ..... C=50, gamma=0.01, kernel=linear, score=0.723, total=   0.0s
[CV] C=50, gamma=0.01, kernel=linear .................................
[CV] ..... C=50, gamma=0.01, kernel=linear, score=0.815, total=   0.1s
[CV] C=50, gamma=0.01, kernel=linear .................................
[CV] ..... C=50, gamma=0.01, kernel=linear, score=0.738, total=   0.1s
[CV] C=50, gamma=0.01, kernel=linear .................................
[CV] ..... C=50, gamma=0.01, kernel=linear, score=0.815, total=   0.1s
[CV] C=50, gamma=0.01, kernel=poly ...................................
[CV] ....... C=50, gamma=0.01, kernel=poly, score=0.527, total=   0.0s
[CV] C=50, gamma=0.01, kernel=poly ...................................
[CV] ....... C=50, gamma=0.01, kernel=poly, score=0.523, total=   0.0s
[CV] C=50, gamma=0.01, kernel=poly ...................................
[CV] ....... C=50, gamma=0.01, kernel=poly, score=0.538, total=   0.0s
[CV] C=50, gamma=0.01, kernel=poly ...................................
[CV] ....... C=50, gamma=0.01, kernel=poly, score=0.515, total=   0.0s
[CV] C=50, gamma=0.01, kernel=poly ...................................
[CV] ....... C=50, gamma=0.01, kernel=poly, score=0.538, total=   0.0s
[CV] C=50, gamma=0.01, kernel=rbf ....................................
[CV] ........ C=50, gamma=0.01, kernel=rbf, score=0.885, total=   0.0s
[CV] C=50, gamma=0.01, kernel=rbf ....................................
[CV] ........ C=50, gamma=0.01, kernel=rbf, score=0.854, total=   0.0s
[CV] C=50, gamma=0.01, kernel=rbf ....................................
[CV] ........ C=50, gamma=0.01, kernel=rbf, score=0.908, total=   0.0s
[CV] C=50, gamma=0.01, kernel=rbf ....................................
[CV] ........ C=50, gamma=0.01, kernel=rbf, score=0.815, total=   0.0s
[CV] C=50, gamma=0.01, kernel=rbf ....................................
[CV] ........ C=50, gamma=0.01, kernel=rbf, score=0.900, total=   0.0s
[CV] C=50, gamma=0.001, kernel=linear ................................
[CV] .... C=50, gamma=0.001, kernel=linear, score=0.771, total=   0.1s
[CV] C=50, gamma=0.001, kernel=linear ................................
[CV] .... C=50, gamma=0.001, kernel=linear, score=0.723, total=   0.1s
[CV] C=50, gamma=0.001, kernel=linear ................................
[CV] .... C=50, gamma=0.001, kernel=linear, score=0.815, total=   0.1s
[CV] C=50, gamma=0.001, kernel=linear ................................
[CV] .... C=50, gamma=0.001, kernel=linear, score=0.738, total=   0.1s
[CV] C=50, gamma=0.001, kernel=linear ................................
[CV] .... C=50, gamma=0.001, kernel=linear, score=0.815, total=   0.0s
[CV] C=50, gamma=0.001, kernel=poly ..................................
[CV] ...... C=50, gamma=0.001, kernel=poly, score=0.504, total=   0.0s
[CV] C=50, gamma=0.001, kernel=poly ..................................
[CV] ...... C=50, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=50, gamma=0.001, kernel=poly ..................................
[CV] ...... C=50, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=50, gamma=0.001, kernel=poly ..................................
[CV] ...... C=50, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=50, gamma=0.001, kernel=poly ..................................
[CV] ...... C=50, gamma=0.001, kernel=poly, score=0.500, total=   0.0s
[CV] C=50, gamma=0.001, kernel=rbf ...................................
[CV] ....... C=50, gamma=0.001, kernel=rbf, score=0.756, total=   0.0s
[CV] C=50, gamma=0.001, kernel=rbf ...................................
[CV] ....... C=50, gamma=0.001, kernel=rbf, score=0.731, total=   0.0s
[CV] C=50, gamma=0.001, kernel=rbf ...................................
[CV] ....... C=50, gamma=0.001, kernel=rbf, score=0.808, total=   0.0s
[CV] C=50, gamma=0.001, kernel=rbf ...................................
[CV] ....... C=50, gamma=0.001, kernel=rbf, score=0.738, total=   0.0s
[CV] C=50, gamma=0.001, kernel=rbf ...................................
[CV] ....... C=50, gamma=0.001, kernel=rbf, score=0.815, total=   0.0s
[CV] C=50, gamma=0.0001, kernel=linear ...............................
[CV] ... C=50, gamma=0.0001, kernel=linear, score=0.771, total=   0.0s
[CV] C=50, gamma=0.0001, kernel=linear ...............................
[CV] ... C=50, gamma=0.0001, kernel=linear, score=0.723, total=   0.1s
[CV] C=50, gamma=0.0001, kernel=linear ...............................
[CV] ... C=50, gamma=0.0001, kernel=linear, score=0.815, total=   0.1s
[CV] C=50, gamma=0.0001, kernel=linear ...............................
[CV] ... C=50, gamma=0.0001, kernel=linear, score=0.738, total=   0.1s
[CV] C=50, gamma=0.0001, kernel=linear ...............................
[CV] ... C=50, gamma=0.0001, kernel=linear, score=0.815, total=   0.1s
[CV] C=50, gamma=0.0001, kernel=poly .................................
[CV] ..... C=50, gamma=0.0001, kernel=poly, score=0.504, total=   0.0s
[CV] C=50, gamma=0.0001, kernel=poly .................................
[CV] ..... C=50, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=50, gamma=0.0001, kernel=poly .................................
[CV] ..... C=50, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=50, gamma=0.0001, kernel=poly .................................
[CV] ..... C=50, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=50, gamma=0.0001, kernel=poly .................................
[CV] ..... C=50, gamma=0.0001, kernel=poly, score=0.500, total=   0.0s
[CV] C=50, gamma=0.0001, kernel=rbf ..................................
[CV] ...... C=50, gamma=0.0001, kernel=rbf, score=0.740, total=   0.0s
[CV] C=50, gamma=0.0001, kernel=rbf ..................................
[CV] ...... C=50, gamma=0.0001, kernel=rbf, score=0.723, total=   0.0s
[CV] C=50, gamma=0.0001, kernel=rbf ..................................
[CV] ...... C=50, gamma=0.0001, kernel=rbf, score=0.731, total=   0.0s
[CV] C=50, gamma=0.0001, kernel=rbf ..................................
[CV] ...... C=50, gamma=0.0001, kernel=rbf, score=0.677, total=   0.0s
[CV] C=50, gamma=0.0001, kernel=rbf ..................................
[CV] ...... C=50, gamma=0.0001, kernel=rbf, score=0.777, total=   0.0s

[Parallel(n_jobs=1)]: Done 525 out of 525 | elapsed:   30.6s finished
Out[35]:
GridSearchCV(cv=None, error_score=nan,
             estimator=SVC(C=1.0, break_ties=False, cache_size=200,
                           class_weight=None, coef0=0.0,
                           decision_function_shape='ovr', degree=3,
                           gamma='scale', kernel='rbf', max_iter=-1,
                           probability=False, random_state=None, shrinking=True,
                           tol=0.001, verbose=False),
             iid='deprecated', n_jobs=None,
             param_grid={'C': [0.01, 0.05, 0.1, 0.5, 1, 10, 50],
                         'gamma': [1, 0.1, 0.01, 0.001, 0.0001],
                         'kernel': ['linear', 'poly', 'rbf']},
             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,
             scoring=None, verbose=3)
In [36]:
grid.best_params_
print("========================================================")
grid.best_score_
print("========================================================")
grid.best_estimator_
Out[36]:
{'C': 10, 'gamma': 0.1, 'kernel': 'rbf'}
========================================================
Out[36]:
0.897075748678802
========================================================
Out[36]:
SVC(C=10, break_ties=False, cache_size=200, class_weight=None, coef0=0.0,
    decision_function_shape='ovr', degree=3, gamma=0.1, kernel='rbf',
    max_iter=-1, probability=False, random_state=None, shrinking=True,
    tol=0.001, verbose=False)
In [37]:
grid_predictions = grid.predict(X_test_pca)
print(confusion_matrix(y_test,grid_predictions))
print(classification_report(y_test,grid_predictions))

[[39  0  0]
 [ 2 84  4]
 [ 0  2 32]]
              precision    recall  f1-score   support

           0       0.95      1.00      0.97        39
           1       0.98      0.93      0.95        90
           2       0.89      0.94      0.91        34

    accuracy                           0.95       163
   macro avg       0.94      0.96      0.95       163
weighted avg       0.95      0.95      0.95       163

SVM Optimized

In [38]:
# Retrain the model with grid search best parameter

svm_opt = SVC(C=10, gamma= 0.1, kernel='rbf')

svm_opt.fit(X_train_pca, y_train)
print ('Score', svm_opt.score(X_test_pca, y_test))
Out[38]:
SVC(C=10, break_ties=False, cache_size=200, class_weight=None, coef0=0.0,
    decision_function_shape='ovr', degree=3, gamma=0.1, kernel='rbf',
    max_iter=-1, probability=False, random_state=None, shrinking=True,
    tol=0.001, verbose=False)
Score 0.950920245398773
In [39]:
 # Cross validation scores
scores = cross_val_score(svm_opt,
                         X_train_pca,
                         y_train,
                         cv=5,
                         scoring='r2')

# Get model accuracy
accuracy = svm_opt.score(X_test_pca, y_test)
accuracy
Out[39]:
0.950920245398773

The performance improved from 84% to 95%

The model is performing well after the PCA dimentionality deduction.